Oct 30 23:53:25.492116 ip-10-0-130-152 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:53:25.905067 ip-10-0-130-152 kubenswrapper[2572]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:25.905067 ip-10-0-130-152 kubenswrapper[2572]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:53:25.905067 ip-10-0-130-152 kubenswrapper[2572]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:25.905067 ip-10-0-130-152 kubenswrapper[2572]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:53:25.911580 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911490 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.911580 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911577 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.911580 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911582 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.911580 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911585 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.911580 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911587 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911591 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911594 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911596 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911599 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911601 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911604 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911607 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911609 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911613 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911615 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911618 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911621 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911623 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911626 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911628 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911631 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911633 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911636 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911638 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.911803 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911641 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911643 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911646 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911649 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911652 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911655 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911657 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911660 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911662 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911665 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911667 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911670 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911672 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911674 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911677 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911680 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911682 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911685 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911687 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911690 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.912257 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911693 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911695 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911697 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911700 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911702 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911705 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911708 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911710 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911713 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911715 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911717 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911720 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911722 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911725 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911728 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911731 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911733 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911736 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911739 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.912719 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911742 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911744 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911747 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911750 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911752 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911755 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911758 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911760 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911763 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911766 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911768 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911771 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911774 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911776 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911796 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911800 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911802 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911805 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911808 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.913210 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911812 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911824 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911829 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911832 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911835 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911837 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911840 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911843 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911845 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911848 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911850 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911853 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911857 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911861 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911864 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911868 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911871 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911874 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911876 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.913649 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911879 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911882 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911884 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911887 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.911890 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912688 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912695 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912698 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912701 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912704 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912706 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912709 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912712 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912714 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912717 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912721 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912724 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912726 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912729 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912732 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.914106 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912734 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912737 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912739 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912742 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912746 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912748 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912751 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912754 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912756 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912759 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912762 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912764 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912767 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912770 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912773 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912776 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912797 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912801 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912803 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.914584 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912806 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912809 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912811 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912814 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912816 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912819 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912822 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912824 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912827 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912831 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912835 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912839 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912842 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912844 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912847 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912849 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912852 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912855 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912858 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.915049 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912860 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912863 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912865 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912868 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912870 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912873 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912875 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912878 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912880 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912883 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912886 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912888 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912891 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912893 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912895 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912898 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912900 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912903 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912905 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912908 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.915508 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912910 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912913 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912916 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912919 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912923 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912925 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912928 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912930 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912934 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912938 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912941 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912944 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912946 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912950 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912954 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912957 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912960 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912963 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912965 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.916019 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912968 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912971 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912973 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912976 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912978 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912980 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912983 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912985 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912988 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912990 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912993 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912995 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.912999 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913002 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913479 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913484 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913488 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913491 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913494 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.916460 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913496 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913499 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913501 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913504 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913507 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913509 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913512 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913514 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913517 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913519 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913521 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913524 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913526 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913529 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913531 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913534 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913536 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913539 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913542 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.916909 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913545 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913547 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913550 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913552 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913555 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913557 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913560 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913562 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913565 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913568 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913571 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913574 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913577 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913580 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913582 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913584 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913587 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913589 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913592 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913595 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.917339 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913597 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913600 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913602 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913604 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913607 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913609 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913611 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913614 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913616 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913619 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913621 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913624 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913627 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913629 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913631 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913634 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913636 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913639 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913641 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913643 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.917812 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913646 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913648 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913651 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913654 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913657 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913660 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913663 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913665 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913669 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913673 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913676 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913679 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913682 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913684 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913687 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913689 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913692 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913695 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913698 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.918268 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913700 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913702 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913705 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913708 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913710 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913712 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913715 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913717 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913720 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913722 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913725 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913727 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913730 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913732 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913735 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913738 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913740 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913743 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913746 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913748 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.918705 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913752 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913756 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.913759 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914691 2572 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914709 2572 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914717 2572 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914722 2572 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914727 2572 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914730 2572 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914735 2572 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914739 2572 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914742 2572 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914745 2572 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914748 2572 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914752 2572 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914755 2572 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914758 2572 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914760 2572 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914763 2572 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914766 2572 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914769 2572 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914773 2572 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914775 2572 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:53:25.919201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914796 2572 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914801 2572 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914804 2572 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914808 2572 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914811 2572 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914815 2572 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914818 2572 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914821 2572 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914825 2572 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914828 2572 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914831 2572 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914834 2572 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914838 2572 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914843 2572 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914846 2572 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914848 2572 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914851 2572 flags.go:64] FLAG: --enable-server="true" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914854 2572 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914861 2572 flags.go:64] FLAG: --event-burst="100" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914864 2572 flags.go:64] FLAG: --event-qps="50" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914867 2572 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914870 2572 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914873 2572 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914877 2572 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:53:25.919720 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914880 2572 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914883 2572 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914886 2572 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914889 2572 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914891 2572 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914894 2572 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914897 2572 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914900 2572 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914903 2572 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914906 2572 flags.go:64] FLAG: --feature-gates="" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914910 2572 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914913 2572 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914916 2572 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914919 2572 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914922 2572 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914925 2572 flags.go:64] FLAG: --help="false" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914928 2572 flags.go:64] FLAG: --hostname-override="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914932 2572 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914935 2572 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914938 2572 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914942 2572 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914946 2572 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914949 2572 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:53:25.920312 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914952 2572 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914955 2572 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914957 2572 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914960 2572 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914963 2572 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914966 2572 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914969 2572 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914972 2572 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914975 2572 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914977 2572 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914980 2572 flags.go:64] FLAG: --lock-file="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914983 2572 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914986 2572 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914989 2572 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914995 2572 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.914997 2572 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915000 2572 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915003 2572 flags.go:64] FLAG: --logging-format="text" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915006 2572 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915009 2572 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915012 2572 flags.go:64] FLAG: --manifest-url="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915015 2572 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915020 2572 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915023 2572 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915027 2572 flags.go:64] FLAG: --max-pods="110" Oct 30 23:53:25.920864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915030 2572 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915032 2572 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915036 2572 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915039 2572 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915042 2572 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915047 2572 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915049 2572 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915057 2572 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915060 2572 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915064 2572 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915067 2572 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915069 2572 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915072 2572 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915075 2572 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915078 2572 flags.go:64] FLAG: --port="10250" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915081 2572 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915084 2572 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0100f67647ce5dcca" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915087 2572 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915090 2572 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915093 2572 flags.go:64] FLAG: --register-node="true" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915096 2572 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915100 2572 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915102 2572 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915105 2572 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915108 2572 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915112 2572 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:53:25.921446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915115 2572 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915119 2572 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915122 2572 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915125 2572 flags.go:64] FLAG: --runonce="false" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915128 2572 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915131 2572 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915134 2572 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915137 2572 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915140 2572 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915143 2572 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915146 2572 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915150 2572 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915161 2572 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915165 2572 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915168 2572 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915171 2572 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915174 2572 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915177 2572 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915180 2572 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915185 2572 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915188 2572 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915190 2572 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915194 2572 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915197 2572 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915200 2572 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:53:25.922066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915203 2572 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915206 2572 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915208 2572 flags.go:64] FLAG: --v="2" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915213 2572 flags.go:64] FLAG: --version="false" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915217 2572 flags.go:64] FLAG: --vmodule="" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915221 2572 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915224 2572 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915373 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915379 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915383 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915385 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915388 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915390 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915393 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915395 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915398 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915400 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915404 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915407 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915410 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915412 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.922679 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915415 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915417 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915420 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915423 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915426 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915428 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915430 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915433 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915435 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915438 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915441 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915443 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915446 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915448 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915451 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915453 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915456 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915458 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915461 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915465 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.923275 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915467 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915470 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915472 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915475 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915477 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915480 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915483 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915486 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915491 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915498 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915502 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915505 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915508 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915510 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915513 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915516 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915518 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915521 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915524 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.923741 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915526 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915529 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915531 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915534 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915536 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915539 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915541 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915544 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915546 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915548 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915551 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915553 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915557 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915560 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915562 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915565 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915567 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915570 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915572 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915575 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.924195 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915577 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915580 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915583 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915593 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915596 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915600 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915603 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915606 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915609 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915612 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915614 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915616 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915619 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915621 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915624 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915626 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915629 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915632 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915634 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.924713 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915636 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915639 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915641 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915644 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915646 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915650 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915653 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915655 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915657 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915660 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915663 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915665 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915667 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915670 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.915696 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915738 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.925231 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915744 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915748 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915751 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915754 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915756 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915759 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915761 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915764 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915767 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915769 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915772 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915774 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915777 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915797 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915800 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915803 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915806 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915808 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915811 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915814 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.925600 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915817 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915820 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915823 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915826 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915828 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915831 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915833 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915836 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915838 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915841 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915843 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915846 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915850 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915852 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915859 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915861 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915864 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915868 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915872 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.926074 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915875 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915878 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915881 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915884 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915887 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915890 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915893 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915896 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915898 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915902 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915905 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915908 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915910 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915913 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915917 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915919 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915922 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915924 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915927 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915929 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.926506 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915931 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915934 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915937 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915939 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915941 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915945 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915948 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915950 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915953 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915956 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915959 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915961 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915964 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915966 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915969 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915972 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915976 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915978 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915981 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.926981 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915984 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915986 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915988 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915991 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915993 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915996 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.915998 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916003 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916005 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916008 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916010 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916013 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916015 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916018 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916020 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916023 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916026 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916028 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916032 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.927431 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916036 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916038 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916040 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916046 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916049 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916051 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916054 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.916056 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.923608 2572 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.923624 2572 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.923635 2572 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.923640 2572 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923700 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923706 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923709 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923712 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923715 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923718 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923720 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923723 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.927884 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923726 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923729 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923731 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923734 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923737 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923739 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923742 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923745 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923747 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923750 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923753 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923756 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923758 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923761 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923763 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923767 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923769 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923772 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923774 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.928349 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923777 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923792 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923796 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923798 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923801 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923804 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923806 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923809 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923811 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923814 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923816 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923818 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923821 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923824 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923828 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923831 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923835 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923837 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923841 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.928876 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923844 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923847 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923850 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923853 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923855 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923858 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923861 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923863 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923865 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923868 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923872 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923874 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923876 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923879 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923881 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923884 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923886 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923889 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923891 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.929324 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923894 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923896 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923899 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923901 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923904 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923906 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923908 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923911 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923913 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923916 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923919 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923921 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923924 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923926 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923929 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923931 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923934 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923936 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923938 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923941 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.929809 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923943 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923946 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923948 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923950 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923953 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923956 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923958 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923961 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923963 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923966 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923968 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923971 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923973 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923975 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923978 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923980 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923983 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923985 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923987 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923990 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.930260 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.923992 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.924018 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924177 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924183 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924186 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924188 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924191 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924194 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924197 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924200 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924202 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924205 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924208 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924210 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924213 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924215 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.930721 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924218 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924221 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924223 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924226 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924229 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924231 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924234 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924236 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924239 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924241 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924244 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924246 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924250 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924254 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924256 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924259 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924261 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924264 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924266 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.931103 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924269 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924271 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924273 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924276 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924278 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924281 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924283 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924285 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924288 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924290 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924293 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924311 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924315 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924318 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924321 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924324 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924327 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924330 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924332 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.931545 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924335 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924337 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924340 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924342 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924345 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924347 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924350 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924352 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924354 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924357 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924359 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924362 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924364 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924366 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924369 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924371 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924374 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924377 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924381 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.932006 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924384 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924387 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924390 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924392 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924394 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924397 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924399 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924402 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924404 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924406 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924409 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924412 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924415 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924421 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924423 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924426 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924428 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924431 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924433 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924436 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.932444 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924439 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924441 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924444 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924446 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924449 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924451 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924453 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924456 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924458 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924461 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924464 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924466 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924468 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924471 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924473 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.924496 2572 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924530 2572 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:53:25.932922 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924534 2572 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924537 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924540 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924543 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924546 2572 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924548 2572 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924551 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924554 2572 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924556 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924561 2572 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924565 2572 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924568 2572 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924571 2572 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924574 2572 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924576 2572 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924578 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924581 2572 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924583 2572 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924586 2572 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:53:25.933307 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924588 2572 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924591 2572 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924593 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924596 2572 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924598 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924601 2572 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924604 2572 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924606 2572 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924609 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924611 2572 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924613 2572 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924616 2572 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924619 2572 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924621 2572 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924624 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924626 2572 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924629 2572 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924631 2572 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924634 2572 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:53:25.933738 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924636 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924639 2572 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924642 2572 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924646 2572 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924649 2572 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924651 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924654 2572 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924656 2572 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924658 2572 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924661 2572 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924663 2572 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924666 2572 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924668 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924670 2572 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924674 2572 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924678 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924680 2572 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924683 2572 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924686 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:53:25.934345 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924688 2572 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924691 2572 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924693 2572 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924696 2572 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924698 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924700 2572 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924703 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924705 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924708 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924711 2572 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924713 2572 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924716 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924719 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924721 2572 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924723 2572 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924726 2572 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924730 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924732 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924735 2572 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924737 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:53:25.935070 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924739 2572 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924742 2572 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924744 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924747 2572 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924749 2572 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924751 2572 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924754 2572 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924756 2572 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924759 2572 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924761 2572 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924764 2572 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924766 2572 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924768 2572 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924771 2572 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924774 2572 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924776 2572 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924778 2572 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924806 2572 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924809 2572 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924811 2572 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:53:25.935747 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924814 2572 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924816 2572 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924819 2572 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924821 2572 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924824 2572 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924826 2572 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924831 2572 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:25.924834 2572 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.924901 2572 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.928150 2572 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir" Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.929194 2572 server.go:1014] "Starting client certificate rotation" Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.929296 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.930146 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:53:25.936258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.931685 2572 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:53:25.936558 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.936335 2572 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:53:25.945802 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.945774 2572 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:53:25.950300 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.950288 2572 log.go:25] "Validated CRI v1 image API" Oct 30 23:53:25.951173 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.951138 2572 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:53:25.953746 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.953726 2572 fs.go:135] Filesystem UUIDs: map[40960bc2-2bb9-4f7c-8263-0e6f0e0e6489:/dev/nvme0n1p4 6beb498b-e7f2-427a-9b79-61dc40a3e89b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 23:53:25.953825 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.953745 2572 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}] Oct 30 23:53:25.959180 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.959064 2572 manager.go:217] Machine: {Timestamp:2026-10-30 23:53:25.957593932 +0000 UTC m=+0.370786005 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3104770 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a041fd69eadefa861aba8e3828ad SystemUUID:ec28a041-fd69-eade-fa86-1aba8e3828ad BootID:d270d178-079e-470b-8da8-9a651a894d7f Filesystems:[{Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:e1:71:e7:eb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:e1:71:e7:eb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:2a:b3:3f:c2:24:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:53:25.959180 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.959179 2572 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:53:25.959272 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.959253 2572 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:53:25.961335 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.961316 2572 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:53:25.961487 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.961339 2572 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-152.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:53:25.962183 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.962173 2572 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:53:25.962214 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.962186 2572 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:53:25.962241 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.962214 2572 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:53:25.962241 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.962226 2572 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:53:25.963490 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.963480 2572 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:53:25.968107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968095 2572 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:53:25.968232 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968223 2572 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:53:25.968632 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968615 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:53:25.968923 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968913 2572 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:53:25.968952 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968930 2572 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:53:25.968952 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968946 2572 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:53:25.969008 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968955 2572 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:53:25.969008 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.968963 2572 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:53:25.969356 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.969344 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:53:25.971938 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.971921 2572 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:53:25.978722 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.978707 2572 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:53:25.979676 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.979664 2572 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:53:25.979715 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.979695 2572 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:53:25.980689 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980678 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980695 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980702 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980708 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980713 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980719 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980726 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:53:25.980730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980732 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:53:25.980909 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980739 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:53:25.980909 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980745 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:53:25.980909 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980753 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:53:25.980909 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.980761 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:53:25.981730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.981687 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:53:25.981861 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.981849 2572 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:53:25.985691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.985670 2572 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-130-152.us-east-2.compute.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope Oct 30 23:53:25.987021 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.987008 2572 server.go:1263] "Started kubelet" Oct 30 23:53:25.987176 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.987137 2572 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:53:25.987310 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.987104 2572 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:53:25.987374 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.987330 2572 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:53:25.987749 ip-10-0-130-152 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:53:25.988472 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:25.988446 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-130-152.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" Oct 30 23:53:25.988584 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:25.988526 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" Oct 30 23:53:25.988584 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.988536 2572 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:53:25.991459 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.991443 2572 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:53:25.993083 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:25.993059 2572 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache" Oct 30 23:53:25.997215 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.997200 2572 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:53:25.997298 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.997225 2572 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:53:25.997298 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:25.997270 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:25.997970 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.997942 2572 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:53:25.997970 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.997972 2572 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:53:25.998161 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.997989 2572 factory.go:55] Registering systemd factory Oct 30 23:53:25.998161 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998003 2572 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:53:25.998161 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998070 2572 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:53:25.998161 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998076 2572 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:53:25.998457 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998235 2572 factory.go:153] Registering CRI-O factory Oct 30 23:53:25.998457 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998247 2572 factory.go:223] Registration of the crio container factory successfully Oct 30 23:53:25.998457 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998284 2572 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:53:25.998457 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998299 2572 factory.go:103] Registering Raw factory Oct 30 23:53:25.998457 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998315 2572 manager.go:1199] Started watching for new ooms in manager Oct 30 23:53:25.998947 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.998929 2572 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:53:25.999348 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:25.998185 2572 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-130-152.us-east-2.compute.internal.18e373ce2d793139 default 0 0001-01-01 00:00:00 +0000 UTC map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-130-152.us-east-2.compute.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-130-152.us-east-2.compute.internal,},FirstTimestamp:2026-10-30 23:53:25.986963769 +0000 UTC m=+0.400155843,LastTimestamp:2026-10-30 23:53:25.986963769 +0000 UTC m=+0.400155843,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-130-152.us-east-2.compute.internal,}" Oct 30 23:53:25.999581 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:25.999568 2572 manager.go:319] Starting recovery of all containers Oct 30 23:53:26.004998 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.004972 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfwqk" Oct 30 23:53:26.007936 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.007875 2572 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-130-152.us-east-2.compute.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms" Oct 30 23:53:26.010873 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.010853 2572 manager.go:324] Recovery completed Oct 30 23:53:26.015211 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.015192 2572 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" Oct 30 23:53:26.015726 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.015714 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.018415 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.018398 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.018509 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.018429 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.018509 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.018445 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.018706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.018683 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-rfwqk" Oct 30 23:53:26.019010 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.018997 2572 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:53:26.019010 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019009 2572 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:53:26.019090 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019024 2572 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:53:26.019090 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019025 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.019686 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019672 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.019749 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019699 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.019749 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.019713 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.022135 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.022123 2572 policy_none.go:50] "Start" Oct 30 23:53:26.022193 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.022139 2572 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:53:26.022193 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.022150 2572 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:53:26.024660 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.024648 2572 policy_none.go:44] "Start" Oct 30 23:53:26.067996 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.067965 2572 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:53:26.070225 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.070212 2572 manager.go:341] "Starting Device Plugin manager" Oct 30 23:53:26.070952 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.070937 2572 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:53:26.071035 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.070955 2572 server.go:87] "Starting device plugin registration server" Oct 30 23:53:26.071218 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.071204 2572 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:53:26.071283 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.071220 2572 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:53:26.071361 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.071339 2572 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:53:26.071455 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.071440 2572 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:53:26.071511 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.071456 2572 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:53:26.071988 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.071968 2572 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:53:26.072044 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.072012 2572 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.137112 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.137083 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:53:26.139131 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.138580 2572 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:53:26.139131 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.138604 2572 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:53:26.139131 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.138653 2572 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:53:26.139131 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.138688 2572 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful" Oct 30 23:53:26.141794 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.141766 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:26.171927 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.171722 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.172762 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.172749 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.172844 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.172794 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.172844 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.172811 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.172844 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.172835 2572 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.180747 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.180729 2572 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.180819 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.180752 2572 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-130-152.us-east-2.compute.internal\": node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.194194 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.194171 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.239690 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.239662 2572 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal"] Oct 30 23:53:26.239816 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.239734 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.243246 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.243225 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.243314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.243263 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.243314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.243278 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.245602 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.245589 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.245756 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.245743 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.245813 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.245770 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.246402 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246383 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.246485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246404 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.246485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246416 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.246485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246424 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.246485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246429 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.246485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.246434 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.248456 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.248442 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.248523 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.248475 2572 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:53:26.249224 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.249209 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:53:26.249291 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.249235 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:53:26.249291 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.249247 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:53:26.277231 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.277215 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.279911 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.279897 2572 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.294484 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.294465 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.395453 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.395418 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.399800 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.399758 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.399868 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.399821 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.399868 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.399843 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.496199 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.496129 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.500465 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.500509 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500479 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.500541 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500508 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.500575 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500538 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.500610 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500570 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.500610 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.500575 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.580651 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.580622 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.583773 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.583745 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:26.596421 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.596402 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.696929 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.696898 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.797475 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.797413 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.897892 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.897859 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:26.929326 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.929295 2572 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials" Oct 30 23:53:26.929774 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:26.929455 2572 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received" Oct 30 23:53:26.997964 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:26.997939 2572 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:53:27.004405 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.004391 2572 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:27.024580 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.024553 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:26 +0000 UTC" deadline="2028-07-12 05:07:57.393686618 +0000 UTC" Oct 30 23:53:27.024580 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.024577 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14885h14m30.369112274s" Oct 30 23:53:27.097677 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.097616 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:27.116560 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.116536 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:53:27.117923 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.117908 2572 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:53:27.127168 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.127136 2572 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:53:27.365053 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:27.365010 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e44186c11950043d5117909b34eae0d.slice/crio-ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5 WatchSource:0}: Error finding container ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5: Status 404 returned error can't find the container with id ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5 Oct 30 23:53:27.365371 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:27.365351 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d885cc10bee944101eff496e23ac512.slice/crio-3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd WatchSource:0}: Error finding container 3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd: Status 404 returned error can't find the container with id 3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd Oct 30 23:53:27.370355 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.370341 2572 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:53:27.429165 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.429135 2572 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:27.497823 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.497777 2572 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:27.970164 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.969850 2572 apiserver.go:52] "Watching apiserver" Oct 30 23:53:27.976665 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.976628 2572 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:53:27.977048 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.977022 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-45ssd","kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal","openshift-cluster-node-tuning-operator/tuned-bsxgt","openshift-dns/node-resolver-q7jkq","kube-system/konnectivity-agent-dbh42","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp","openshift-multus/multus-additional-cni-plugins-n72w7","openshift-multus/multus-rblpn","openshift-image-registry/node-ca-4qqrz","openshift-multus/network-metrics-daemon-6dpdv","openshift-network-diagnostics/network-check-target-8nvcl","openshift-network-operator/iptables-alerter-dx424","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal","openshift-ovn-kubernetes/ovnkube-node-lqv9d"] Oct 30 23:53:27.980137 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.980113 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:27.982542 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.982516 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:27.982639 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:27.982605 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:27.984872 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.984833 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:27.987304 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.987284 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:27.990197 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.990179 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:27.991290 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991272 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991463 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991541 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991484 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991540 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991619 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:53:27.993254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.991635 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:53:27.994167 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.994098 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:27.996677 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.996333 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:27.996769 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.996690 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:53:27.996769 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.996748 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:53:27.996769 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.996746 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:53:27.997951 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:27.997933 2572 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving" Oct 30 23:53:28.001136 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.001114 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.003129 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.002770 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.005327 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.005303 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.005434 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.005389 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:28.008340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008309 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008438 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008366 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008438 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008404 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58qmb\" (UniqueName: \"kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008546 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008441 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.008546 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008473 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.008546 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008500 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.008546 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008537 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008568 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008596 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008625 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b787\" (UniqueName: \"kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008646 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008666 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008686 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008714 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008735 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.008751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008751 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008798 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008826 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008854 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008900 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008934 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9n6jm\" (UniqueName: \"kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.008967 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009004 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009037 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009064 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009069 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.009120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009099 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7ctp6\" (UniqueName: \"kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009135 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.009141 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009169 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009201 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009371 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.009499 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009408 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009504 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009538 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009591 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009629 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009679 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.009752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009721 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009764 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009803 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009937 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.009984 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010016 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010048 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.010083 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010081 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010114 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010137 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010182 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010230 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010316 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010403 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010427 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8z54h\" (UniqueName: \"kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010445 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.010477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010467 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjbdf\" (UniqueName: \"kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.011019 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010524 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.011019 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.010546 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.014069 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.012405 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.058218 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.058189 2572 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162" Oct 30 23:53:28.058342 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.058300 2572 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdtmc" Oct 30 23:53:28.081374 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.081349 2572 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wdtmc" Oct 30 23:53:28.097544 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097512 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:53:28.097682 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097601 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:53:28.097682 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097646 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:53:28.097682 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097660 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:53:28.097860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097718 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:53:28.097860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097724 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:53:28.097860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097780 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:53:28.097860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097809 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:53:28.097860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097858 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:53:28.098107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097900 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:53:28.098107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097931 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:53:28.098107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097946 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:53:28.098107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.097996 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:53:28.098107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.098057 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:53:28.098394 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.098376 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:53:28.098893 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.098819 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:53:28.099065 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099030 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:53:28.099175 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099102 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\"" Oct 30 23:53:28.099234 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099200 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\"" Oct 30 23:53:28.099345 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099326 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\"" Oct 30 23:53:28.099398 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099374 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\"" Oct 30 23:53:28.099454 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099416 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:53:28.099506 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099484 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\"" Oct 30 23:53:28.099554 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099528 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\"" Oct 30 23:53:28.099604 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099582 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\"" Oct 30 23:53:28.099656 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.099648 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\"" Oct 30 23:53:28.100700 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.100680 2572 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:53:28.111251 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111222 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.111356 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111263 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.111356 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.111356 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111300 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.111356 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111386 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111394 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111456 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111480 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111532 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111538 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111577 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111592 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111598 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111631 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111632 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111656 2572 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111710 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111742 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111786 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111815 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111828 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111923 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.111959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111954 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.111990 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112008 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112094 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112125 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112231 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112279 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112323 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112341 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112379 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112397 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112413 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112429 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z54h\" (UniqueName: \"kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112443 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112472 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbdf\" (UniqueName: \"kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.112655 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112488 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112525 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112630 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112924 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.112952 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113042 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113038 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113218 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113220 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113262 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113308 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113327 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113361 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113379 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113418 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113456 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.113537 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113521 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113566 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113594 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113636 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113663 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113709 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qmb\" (UniqueName: \"kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113736 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113751 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113790 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113870 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113929 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.113964 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114001 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114031 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b787\" (UniqueName: \"kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.114436 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114092 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114135 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114161 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114169 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114223 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114251 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114276 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114303 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114393 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114401 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114421 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cb42l\" (UniqueName: \"kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114457 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114486 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114490 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114560 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114577 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.115314 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114588 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114607 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114676 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114699 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114716 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114732 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114781 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114813 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6jm\" (UniqueName: \"kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114865 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114898 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.114951 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115007 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115038 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115069 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115083 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.116446 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115097 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nd8n\" (UniqueName: \"kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115070 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115113 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115126 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115166 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115198 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115203 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctp6\" (UniqueName: \"kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115238 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzcpr\" (UniqueName: \"kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115269 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115239 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115320 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115349 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115373 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.115447 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.116033 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.116156 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.117340 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.117329 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.118015 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.117846 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.118449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.118427 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.118768 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.118742 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.141115 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.141085 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z54h\" (UniqueName: \"kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.143431 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.143409 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b787\" (UniqueName: \"kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.144067 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.144037 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qmb\" (UniqueName: \"kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.144835 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.144815 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:28.144835 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.144837 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:28.144835 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.144853 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:28.145048 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.144967 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:28.644946044 +0000 UTC m=+3.058138129 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:28.147053 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.146999 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"4e44186c11950043d5117909b34eae0d","Type":"ContainerStarted","Data":"ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5"} Oct 30 23:53:28.148154 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.148129 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctp6\" (UniqueName: \"kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.148322 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.148301 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6jm\" (UniqueName: \"kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.149119 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.149093 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbdf\" (UniqueName: \"kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.149728 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.149700 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerStarted","Data":"3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd"} Oct 30 23:53:28.203674 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.203579 2572 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:53:28.215954 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.215907 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.215959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.215989 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216014 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216041 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216071 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.216102 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216101 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216130 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216156 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216181 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216208 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216241 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216302 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216282 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216306 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216328 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb42l\" (UniqueName: \"kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216346 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216361 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216376 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216395 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216415 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216430 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nd8n\" (UniqueName: \"kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216444 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216460 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcpr\" (UniqueName: \"kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.216482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216475 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.216811 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.216535 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217165 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.217145 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.217241 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217324 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.217310 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217375 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.217360 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217432 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.217404 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.217566 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.217528 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:28.217617 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.217601 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:28.717579996 +0000 UTC m=+3.130772075 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:28.218193 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218165 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.218315 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218177 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218315 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218248 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218315 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218284 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218315 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.218293 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:28.218538 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.218364 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:28.718346034 +0000 UTC m=+3.131538113 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:28.218538 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218418 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218538 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218477 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218542 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218580 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.218691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218620 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.218691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218589 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.218691 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.218675 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.219053 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.219004 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.219164 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.219139 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.234119 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.234089 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb42l\" (UniqueName: \"kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.234777 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.234754 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcpr\" (UniqueName: \"kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.238206 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.238177 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nd8n\" (UniqueName: \"kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.316798 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.316748 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:53:28.317872 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.317846 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:53:28.321770 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.321749 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:53:28.327032 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.326912 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:28.331198 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.331173 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8411675f_2736_483b_8a74_a15c12287ea1.slice/crio-d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68 WatchSource:0}: Error finding container d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68: Status 404 returned error can't find the container with id d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68 Oct 30 23:53:28.336441 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.336272 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb71f4919_41b2_469c_af26_b016813419ec.slice/crio-374e166122145fe00e542d8545e075785a187c15c8a997a7a7c5f1bc3090e678 WatchSource:0}: Error finding container 374e166122145fe00e542d8545e075785a187c15c8a997a7a7c5f1bc3090e678: Status 404 returned error can't find the container with id 374e166122145fe00e542d8545e075785a187c15c8a997a7a7c5f1bc3090e678 Oct 30 23:53:28.339740 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.339657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:53:28.346066 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.346046 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:53:28.351890 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.351833 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode88e13a1_2eda_4ec4_968f_0bada5cf17d9.slice/crio-6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f WatchSource:0}: Error finding container 6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f: Status 404 returned error can't find the container with id 6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f Oct 30 23:53:28.356487 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.356456 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod827a07c8_e408_4526_a58b_4f7982a7dcaa.slice/crio-d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10 WatchSource:0}: Error finding container d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10: Status 404 returned error can't find the container with id d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10 Oct 30 23:53:28.370887 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.370776 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-rblpn" Oct 30 23:53:28.374723 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.374564 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:28.377087 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.377067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:53:28.382398 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.382369 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44459cea_2fa8_44fb_ad9f_9aeafcdae6df.slice/crio-eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c WatchSource:0}: Error finding container eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c: Status 404 returned error can't find the container with id eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c Oct 30 23:53:28.387675 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.387649 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode7bfb22a_c8c3_4051_8f4b_8a2527fc44ae.slice/crio-4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665 WatchSource:0}: Error finding container 4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665: Status 404 returned error can't find the container with id 4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665 Oct 30 23:53:28.390684 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:53:28.390659 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod721668b5_4594_43bb_82ae_f8fd5e80fd2e.slice/crio-c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa WatchSource:0}: Error finding container c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa: Status 404 returned error can't find the container with id c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa Oct 30 23:53:28.720456 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.720363 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:28.720456 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.720431 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:28.720464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720541 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720551 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720609 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:29.720591552 +0000 UTC m=+4.133783614 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720630 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:29.720620848 +0000 UTC m=+4.133812911 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720642 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720673 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:28.720719 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720686 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:28.721105 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:28.720750 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:29.720732833 +0000 UTC m=+4.133924921 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:29.085089 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.084852 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:28 +0000 UTC" deadline="2028-03-28 01:35:16.036577537 +0000 UTC" Oct 30 23:53:29.085089 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.084924 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12337h41m46.951660182s" Oct 30 23:53:29.139563 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.138889 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:29.139563 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.139022 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:29.139563 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.139111 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:29.139563 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.139198 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:29.163644 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.163508 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665"} Oct 30 23:53:29.196704 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.196662 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rblpn" event={"ID":"44459cea-2fa8-44fb-ad9f-9aeafcdae6df","Type":"ContainerStarted","Data":"eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c"} Oct 30 23:53:29.214734 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.214650 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7jkq" event={"ID":"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d","Type":"ContainerStarted","Data":"2d8e482a7489e7c1d8bb25e103ce9c5a4d63006df4754b09107af3a7e6adb454"} Oct 30 23:53:29.228435 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.228334 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerStarted","Data":"d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68"} Oct 30 23:53:29.237869 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.237760 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx424" event={"ID":"721668b5-4594-43bb-82ae-f8fd5e80fd2e","Type":"ContainerStarted","Data":"c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa"} Oct 30 23:53:29.239908 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.239851 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qqrz" event={"ID":"827a07c8-e408-4526-a58b-4f7982a7dcaa","Type":"ContainerStarted","Data":"d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10"} Oct 30 23:53:29.252116 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.252085 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" event={"ID":"e88e13a1-2eda-4ec4-968f-0bada5cf17d9","Type":"ContainerStarted","Data":"6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f"} Oct 30 23:53:29.265577 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.265543 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbh42" event={"ID":"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296","Type":"ContainerStarted","Data":"c97249d546434a48bb4612f4f2e8ba46b234fd76e9e7f7a22671035cc42a440d"} Oct 30 23:53:29.268801 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.268758 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" event={"ID":"b71f4919-41b2-469c-af26-b016813419ec","Type":"ContainerStarted","Data":"374e166122145fe00e542d8545e075785a187c15c8a997a7a7c5f1bc3090e678"} Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.732887 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.733173 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:29.733209 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733356 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733415 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:31.733397478 +0000 UTC m=+6.146589554 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733473 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733506 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:31.733495662 +0000 UTC m=+6.146687738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733578 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733592 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733604 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:29.734039 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:29.733633 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:31.733624099 +0000 UTC m=+6.146816174 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:30.085901 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.085835 2572 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:28 +0000 UTC" deadline="2028-06-06 06:31:32.759502747 +0000 UTC" Oct 30 23:53:30.085901 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.085898 2572 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14022h38m2.673608054s" Oct 30 23:53:30.143077 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.142479 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:30.143077 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:30.142671 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:30.281678 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.281594 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"4e44186c11950043d5117909b34eae0d","Type":"ContainerStarted","Data":"c876dcc48c64642c4cd589acaea644f9a8a615e87b779b3ebab7e4f7377859a8"} Oct 30 23:53:30.287719 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.287690 2572 generic.go:356] "Generic (PLEG): container finished" podID="9d885cc10bee944101eff496e23ac512" containerID="dca40c86edaf5f621b4245ea00d4d7fe5f36c55cb96bea083d204c671602b52e" exitCode=0 Oct 30 23:53:30.287893 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.287759 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerDied","Data":"dca40c86edaf5f621b4245ea00d4d7fe5f36c55cb96bea083d204c671602b52e"} Oct 30 23:53:30.314807 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:30.314739 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" podStartSLOduration=3.314721124 podStartE2EDuration="3.314721124s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:30.299800581 +0000 UTC m=+4.712992660" watchObservedRunningTime="2026-10-30 23:53:30.314721124 +0000 UTC m=+4.727913206" Oct 30 23:53:31.139371 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.139319 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:31.139881 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.139480 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:31.139969 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.139890 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:31.140029 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.139996 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:31.297558 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.297032 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerStarted","Data":"938af9deaef2c3605938b8cc551be2e8a315350d09687ae9fb64fecbab6089c7"} Oct 30 23:53:31.756735 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.756650 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:31.756735 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.756731 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:31.756933 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:31.756765 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:31.756933 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.756919 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:31.756992 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.756981 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:35.756964749 +0000 UTC m=+10.170156834 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:31.757077 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757060 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:31.757146 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757133 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:35.757118222 +0000 UTC m=+10.170310297 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:31.757642 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757244 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:31.757642 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757262 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:31.757642 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757274 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:31.757642 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:31.757313 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:35.757299344 +0000 UTC m=+10.170491423 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:32.142614 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:32.141947 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:32.142614 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:32.142094 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:32.302142 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:32.302075 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="964eab2561fdd6c4b1534d83fefb94bd281e960a374a92a30163edc6eb76a677" exitCode=0 Oct 30 23:53:32.303309 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:32.303281 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"964eab2561fdd6c4b1534d83fefb94bd281e960a374a92a30163edc6eb76a677"} Oct 30 23:53:32.329414 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:32.328852 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" podStartSLOduration=5.328834759 podStartE2EDuration="5.328834759s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:31.314996397 +0000 UTC m=+5.728188481" watchObservedRunningTime="2026-10-30 23:53:32.328834759 +0000 UTC m=+6.742026845" Oct 30 23:53:33.141859 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:33.139721 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:33.141859 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:33.139880 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:33.141859 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:33.140196 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:33.141859 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:33.140256 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:33.310004 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:33.309939 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-4qqrz" event={"ID":"827a07c8-e408-4526-a58b-4f7982a7dcaa","Type":"ContainerStarted","Data":"0ca6294691a1839ebbccdb1a79fd21299394650f3694351a41a20ebf771e3b91"} Oct 30 23:53:33.313768 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:33.313702 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-dbh42" event={"ID":"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296","Type":"ContainerStarted","Data":"d55ef091daa8e6b7eb64b116d7ed1badb4e1475125d6c8c4ea83eaf9040d5ad0"} Oct 30 23:53:33.341294 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:33.341225 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-4qqrz" podStartSLOduration=2.632633141 podStartE2EDuration="7.341206036s" totalImagesPullingTime="4.708572895s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:33.325935553 +0000 UTC m=+7.739127637" watchObservedRunningTime="2026-10-30 23:53:33.341206036 +0000 UTC m=+7.754398119" Oct 30 23:53:34.141359 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:34.141088 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:34.141359 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:34.141217 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:34.322679 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:34.322634 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" event={"ID":"e88e13a1-2eda-4ec4-968f-0bada5cf17d9","Type":"ContainerStarted","Data":"98a67698c701091644d2758e64c4268d1285f9e511178edb8675e08f98e78d3f"} Oct 30 23:53:35.139833 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.139733 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:35.139997 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.139917 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:35.140084 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.140056 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:35.140236 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.140182 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:35.787885 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.787850 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:35.788301 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.787908 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:35.788301 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.787959 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:35.788301 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788084 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:35.788301 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788140 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.788124215 +0000 UTC m=+18.201316298 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:35.788531 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788503 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:35.788531 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788516 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:35.788531 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788529 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:35.788675 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788574 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.788560672 +0000 UTC m=+18.201752751 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:35.788675 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788616 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:35.788675 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:35.788659 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:43.788647515 +0000 UTC m=+18.201839590 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:35.831745 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.831715 2572 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:53:35.887075 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.887043 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:35.888259 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.887797 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:35.906107 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:35.906025 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-dbh42" podStartSLOduration=6.077243478 podStartE2EDuration="9.906012694s" totalImagesPullingTime="3.828769216s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:33.341555654 +0000 UTC m=+7.754747738" watchObservedRunningTime="2026-10-30 23:53:35.906012694 +0000 UTC m=+10.319204777" Oct 30 23:53:36.081616 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.080974 2572 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:53:35.831741222Z","UUID":"72ed1e00-5c5f-4d1d-8f9b-b9769946afb9","Handler":null,"Name":"","Endpoint":""} Oct 30 23:53:36.088135 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.087508 2572 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:53:36.088135 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.087542 2572 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:53:36.140625 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.140597 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:36.140808 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:36.140705 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:36.328825 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.328733 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-rblpn" event={"ID":"44459cea-2fa8-44fb-ad9f-9aeafcdae6df","Type":"ContainerStarted","Data":"d9cbf729bce07811f3c8dd04c7347f474effd5b59e826c402c89fe22a905aee0"} Oct 30 23:53:36.330915 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:36.330883 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" event={"ID":"e88e13a1-2eda-4ec4-968f-0bada5cf17d9","Type":"ContainerStarted","Data":"9ea4bc542d24822dba863fbef4b79b4a7347a275b550e13b755061d76bebd778"} Oct 30 23:53:37.139915 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.139840 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:37.140334 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.139848 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:37.140334 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:37.139965 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:37.140334 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:37.140022 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:37.336005 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.335964 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" event={"ID":"e88e13a1-2eda-4ec4-968f-0bada5cf17d9","Type":"ContainerStarted","Data":"0abb5238f007c12374343aaadc8c6fda2f7b7d250a1a2b8e89d6630e0cd5e678"} Oct 30 23:53:37.337675 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.337645 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-q7jkq" event={"ID":"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d","Type":"ContainerStarted","Data":"a6243f0b87c658943cbdca97b5460acb757ebfa14558f6b150a165f33f999468"} Oct 30 23:53:37.340017 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.339989 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="7b4628bdaff3a1a5c2c0ce818cb518a1219c888f9a4e45dd4403761d8c77082a" exitCode=0 Oct 30 23:53:37.340124 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.340080 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"7b4628bdaff3a1a5c2c0ce818cb518a1219c888f9a4e45dd4403761d8c77082a"} Oct 30 23:53:37.355115 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.355070 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-rblpn" podStartSLOduration=3.816800345 podStartE2EDuration="11.355053972s" totalImagesPullingTime="7.538253627s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:36.350757623 +0000 UTC m=+10.763949706" watchObservedRunningTime="2026-10-30 23:53:37.355053972 +0000 UTC m=+11.768246057" Oct 30 23:53:37.369960 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.369918 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" podStartSLOduration=3.303437281 podStartE2EDuration="11.369905898s" totalImagesPullingTime="8.066468617s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:37.354894546 +0000 UTC m=+11.768086630" watchObservedRunningTime="2026-10-30 23:53:37.369905898 +0000 UTC m=+11.783097983" Oct 30 23:53:37.398021 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:37.397927 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-q7jkq" podStartSLOduration=3.351840856 podStartE2EDuration="11.397910321s" totalImagesPullingTime="8.046069465s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:37.370873271 +0000 UTC m=+11.784065355" watchObservedRunningTime="2026-10-30 23:53:37.397910321 +0000 UTC m=+11.811102407" Oct 30 23:53:38.141939 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.141681 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:38.142238 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:38.142061 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:38.343954 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.343856 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" event={"ID":"b71f4919-41b2-469c-af26-b016813419ec","Type":"ContainerStarted","Data":"76484b7b4cef9331de902cffd5fa2c4bb6b4b6bba3ff694928136dba2fd739f1"} Oct 30 23:53:38.349839 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.349807 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="77edef1f8157f3a306bddba181df12a20b81e9adea7d47a53fadc8e05da695d2" exitCode=0 Oct 30 23:53:38.350006 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.349898 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"77edef1f8157f3a306bddba181df12a20b81e9adea7d47a53fadc8e05da695d2"} Oct 30 23:53:38.352146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.352123 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dx424" event={"ID":"721668b5-4594-43bb-82ae-f8fd5e80fd2e","Type":"ContainerStarted","Data":"1f783b7c2e1fa64e87e9ce79ea311b43353ed1e1e343dbc6a8007955a881adba"} Oct 30 23:53:38.363928 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.363890 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" podStartSLOduration=2.624823551 podStartE2EDuration="12.363879823s" totalImagesPullingTime="9.739056272s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:38.363442871 +0000 UTC m=+12.776634955" watchObservedRunningTime="2026-10-30 23:53:38.363879823 +0000 UTC m=+12.777071906" Oct 30 23:53:38.380605 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:38.380556 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dx424" podStartSLOduration=4.697728167 podStartE2EDuration="12.380539759s" totalImagesPullingTime="7.682811592s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:38.37992057 +0000 UTC m=+12.793112653" watchObservedRunningTime="2026-10-30 23:53:38.380539759 +0000 UTC m=+12.793731843" Oct 30 23:53:39.139274 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:39.139239 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:39.139479 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:39.139277 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:39.139479 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:39.139352 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:39.139567 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:39.139489 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:40.141585 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:40.141545 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:40.142193 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:40.141670 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:40.359132 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:40.359096 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="54d98417d2585dce8f2c3f4b030228e724968750597f9c4c5937065765f2d391" exitCode=0 Oct 30 23:53:40.359284 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:40.359148 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"54d98417d2585dce8f2c3f4b030228e724968750597f9c4c5937065765f2d391"} Oct 30 23:53:41.139814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:41.139541 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:41.139814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:41.139591 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:41.139814 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:41.139703 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:41.140434 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:41.140405 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:42.139948 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:42.139913 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:42.140661 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:42.140050 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:42.366530 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:42.366501 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerStarted","Data":"5d2d8465e4d0a0902411fc54462051c75569b569952cfeb00494cc93d47e7c0d"} Oct 30 23:53:43.139126 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.139090 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:43.139289 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.139212 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:43.139514 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.139500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:43.139618 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.139595 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:43.371182 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.371148 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="5d2d8465e4d0a0902411fc54462051c75569b569952cfeb00494cc93d47e7c0d" exitCode=0 Oct 30 23:53:43.371534 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.371216 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"5d2d8465e4d0a0902411fc54462051c75569b569952cfeb00494cc93d47e7c0d"} Oct 30 23:53:43.848858 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.848654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:43.848858 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.848844 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:43.848858 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.848863 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.848872 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.848871 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:43.848903 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.848916 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:53:59.848903854 +0000 UTC m=+34.262095916 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.848963 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.849009 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:59.848997641 +0000 UTC m=+34.262189701 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.849022 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:43.849099 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:43.849063 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:53:59.849052075 +0000 UTC m=+34.262244136 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:44.139062 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.139039 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:44.139151 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:44.139136 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:44.376680 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376646 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"62183501a9009f4adc2ac3583161d3f6410b3b04e53d612e1f8718936dcb1018"} Oct 30 23:53:44.377187 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376736 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"dd2e0058397768497a56b85f4c81613793d6510d52d693452cf8dde539879b05"} Oct 30 23:53:44.377187 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376807 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"e433dd33a9edd90918d061e62a94a53f45b9cc36532de8e7051b27778a3b5ce0"} Oct 30 23:53:44.377187 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376856 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"97cfa28c1037705f2bd228eacf926435c0e54570a47cbd1f52ff11bc9118889e"} Oct 30 23:53:44.377187 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376894 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"2969a4c96096ad281319db7a27888a4a7d83663d2985cf4e7bf7f1f1f841463a"} Oct 30 23:53:44.377187 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.376928 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"91ff840f51441f93bb17f4955dee73b47ecb7b70fe71101efe9fb4824c5380e1"} Oct 30 23:53:44.379633 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.379610 2572 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="8b6206bc0f477477ad1ad576ca2bd0e442a33cf0991645dc055739e4c9b80303" exitCode=0 Oct 30 23:53:44.379719 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:44.379653 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerDied","Data":"8b6206bc0f477477ad1ad576ca2bd0e442a33cf0991645dc055739e4c9b80303"} Oct 30 23:53:45.139140 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:45.138941 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:45.139309 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:45.138972 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:45.139309 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:45.139208 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:45.139421 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:45.139312 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:45.384751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:45.384716 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-n72w7" event={"ID":"8411675f-2736-483b-8a74-a15c12287ea1","Type":"ContainerStarted","Data":"38574f4cb4dee8b5eb2b3155423a15e3164cd52bc16d6cbc4b7cc384a64f3f29"} Oct 30 23:53:45.408029 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:45.407906 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-n72w7" podStartSLOduration=8.784572533 podStartE2EDuration="19.407893288s" totalImagesPullingTime="10.623320755s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:45.407114382 +0000 UTC m=+19.820306464" watchObservedRunningTime="2026-10-30 23:53:45.407893288 +0000 UTC m=+19.821085370" Oct 30 23:53:46.140299 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:46.140266 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:46.140462 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:46.140360 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:46.391202 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:46.391125 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"82f7d41d11c2a21fcee998767f4cbe1166123c520cb47e5a67916d2e05eccc19"} Oct 30 23:53:47.138904 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:47.138866 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:47.138904 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:47.138910 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:47.139098 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:47.139000 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:47.139140 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:47.139116 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:47.972433 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:47.972397 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:47.972922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:47.972525 2572 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:53:47.972960 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:47.972934 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:53:48.139719 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:48.139684 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:48.139966 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:48.139803 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:49.139535 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.139500 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:49.140226 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:49.139601 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:49.140226 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.139654 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:49.140226 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:49.139744 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:49.401021 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.400946 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" event={"ID":"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae","Type":"ContainerStarted","Data":"538799c87eec5d30959337afa7c2bb86e1f2fe98ef6d9caea8c83d19d64eec6b"} Oct 30 23:53:49.401180 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.401165 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:49.417158 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.417136 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:49.444146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:49.444105 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" podStartSLOduration=7.925429748 podStartE2EDuration="23.444093133s" totalImagesPullingTime="15.518663385s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:53:49.443850507 +0000 UTC m=+23.857042589" watchObservedRunningTime="2026-10-30 23:53:49.444093133 +0000 UTC m=+23.857285216" Oct 30 23:53:50.139364 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.139331 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:50.139522 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:50.139448 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:50.270275 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.270242 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-45ssd"] Oct 30 23:53:50.270964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.270938 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:50.271089 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:50.271066 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:50.276070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.276000 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nvcl"] Oct 30 23:53:50.276819 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.276772 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6dpdv"] Oct 30 23:53:50.276919 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.276893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:50.277025 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:50.277005 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:50.403331 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.403256 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:50.403478 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:50.403354 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:50.403754 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.403730 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:50.403887 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.403821 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:50.419602 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.419581 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:53:50.966616 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:50.966451 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7jkq_78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/dns-node-resolver/0.log" Oct 30 23:53:51.546473 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:51.546448 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qqrz_827a07c8-e408-4526-a58b-4f7982a7dcaa/node-ca/0.log" Oct 30 23:53:52.139519 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:52.139487 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:52.139706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:52.139543 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:52.139706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:52.139560 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:52.139706 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:52.139625 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:52.139855 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:52.139729 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:52.139855 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:52.139834 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:54.139597 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:54.139561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:54.140064 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:54.139561 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:54.140064 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:54.139673 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:54.140064 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:54.139829 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:54.140064 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:54.139867 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:54.140064 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:54.139962 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:56.140530 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:56.140496 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:56.141097 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:56.140623 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:56.141097 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:56.140640 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:56.141097 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:56.140705 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:56.141097 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:56.140752 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:56.141097 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:56.140698 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:58.139419 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:58.139384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:58.139958 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:58.139385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:58.139958 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:58.139489 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:53:58.139958 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:58.139385 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:58.139958 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:58.139572 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:53:58.139958 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:58.139636 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:53:59.863172 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:59.863131 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:53:59.863172 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:59.863180 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:53:59.863207 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863290 2572 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863295 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863303 2572 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863319 2572 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863329 2572 projected.go:196] Error preparing data for projected volume kube-api-access-tkvgs for pod openshift-network-diagnostics/network-check-target-8nvcl: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863340 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret podName:4285655f-2bda-4d2b-87f5-b9506c3fdb51 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:31.863327618 +0000 UTC m=+66.276519679 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret") pod "global-pull-secret-syncer-45ssd" (UID: "4285655f-2bda-4d2b-87f5-b9506c3fdb51") : object "kube-system"/"original-pull-secret" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863369 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs podName:4d1f10bd-b2d0-486d-9caa-21b650348f93 nodeName:}" failed. No retries permitted until 2026-10-30 23:54:31.863351124 +0000 UTC m=+66.276543201 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs") pod "network-metrics-daemon-6dpdv" (UID: "4d1f10bd-b2d0-486d-9caa-21b650348f93") : object "openshift-multus"/"metrics-daemon-secret" not registered Oct 30 23:53:59.863740 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:53:59.863387 2572 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs podName:fa89d9ae-29cc-4664-ae9d-ca28b785c7bd nodeName:}" failed. No retries permitted until 2026-10-30 23:54:31.863378378 +0000 UTC m=+66.276570440 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-tkvgs" (UniqueName: "kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs") pod "network-check-target-8nvcl" (UID: "fa89d9ae-29cc-4664-ae9d-ca28b785c7bd") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered] Oct 30 23:54:00.139892 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:00.139808 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:00.139892 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:00.139844 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:00.140098 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:00.139936 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:54:00.140098 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:00.140010 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:54:00.140098 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:00.140040 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:00.140241 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:00.140108 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:54:02.139058 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:02.139018 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:02.139477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:02.139022 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:02.139477 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:02.139136 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:54:02.139477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:02.139023 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:02.139477 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:02.139219 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:54:02.139477 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:02.139268 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:54:04.139101 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:04.139067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:04.139101 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:04.139067 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:04.139565 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:04.139222 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:54:04.139565 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:04.139254 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:04.139565 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:04.139354 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:54:04.139565 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:04.139439 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:54:06.139542 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:06.139514 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:06.140049 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:06.139603 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:54:06.140049 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:06.139669 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:06.140049 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:06.139763 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:54:06.140049 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:06.139820 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:06.140049 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:06.139881 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:54:08.139689 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.139657 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:08.140120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.139660 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:08.140120 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:08.139758 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-45ssd" podUID="4285655f-2bda-4d2b-87f5-b9506c3fdb51" Oct 30 23:54:08.140120 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.139795 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:08.140120 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:08.139868 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-6dpdv" podUID="4d1f10bd-b2d0-486d-9caa-21b650348f93" Oct 30 23:54:08.140120 ip-10-0-130-152 kubenswrapper[2572]: E1030 23:54:08.139928 2572 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-8nvcl" podUID="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" Oct 30 23:54:08.399301 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.399225 2572 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeReady" Oct 30 23:54:08.399455 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.399349 2572 kubelet_node_status.go:477] "Fast updating node status as it just became ready" Oct 30 23:54:08.460068 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.460044 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-dw7gl"] Oct 30 23:54:08.476798 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.476752 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-vscwx"] Oct 30 23:54:08.476924 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.476867 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.487344 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.487326 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:54:08.487579 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.487563 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:54:08.487666 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.487646 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\"" Oct 30 23:54:08.494987 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.494967 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dw7gl"] Oct 30 23:54:08.495063 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.495009 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vscwx"] Oct 30 23:54:08.495063 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.495052 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.499589 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.499566 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-5sqx2"] Oct 30 23:54:08.514307 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.514287 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5sqx2"] Oct 30 23:54:08.514402 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.514384 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.587882 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.587852 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:54:08.588033 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.587905 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:54:08.588033 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.587928 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:54:08.588146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588114 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\"" Oct 30 23:54:08.588146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588126 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:54:08.588146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588140 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:54:08.588254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588202 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:54:08.588254 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588226 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:54:08.588317 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.588294 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\"" Oct 30 23:54:08.625504 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625475 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.625598 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625511 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.625598 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625545 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.625682 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625653 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8cgn\" (UniqueName: \"kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.625726 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625707 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g25jl\" (UniqueName: \"kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.625776 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625723 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.625776 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625745 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd7df\" (UniqueName: \"kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.625776 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625767 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.625917 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.625917 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.625892 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.626034 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.626017 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.726921 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726833 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.726921 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.726921 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726902 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.727169 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726935 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cgn\" (UniqueName: \"kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727169 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726974 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25jl\" (UniqueName: \"kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.727169 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.726995 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727169 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727019 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7df\" (UniqueName: \"kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.727169 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727086 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727364 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727146 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727420 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727345 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727398 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.727477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727464 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.727586 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727516 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.727640 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727602 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.727686 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.727675 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.728033 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.728015 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.730664 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.730642 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.730752 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.730650 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.730813 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.730756 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.751258 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.751229 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7df\" (UniqueName: \"kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:08.751683 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.751660 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25jl\" (UniqueName: \"kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.754525 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.754499 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cgn\" (UniqueName: \"kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.821513 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.821482 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:08.868198 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.868173 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:08.890274 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:08.890253 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:09.174354 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.174319 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-dw7gl"] Oct 30 23:54:09.177828 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:09.177774 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14e77ba7_07ab_4933_9268_760aa9712cb8.slice/crio-47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9 WatchSource:0}: Error finding container 47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9: Status 404 returned error can't find the container with id 47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9 Oct 30 23:54:09.178005 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.177979 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-vscwx"] Oct 30 23:54:09.183311 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:09.183286 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58a99b61_5c4d_41c8_9b7a_172aa3edc72b.slice/crio-fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629 WatchSource:0}: Error finding container fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629: Status 404 returned error can't find the container with id fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629 Oct 30 23:54:09.208299 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.208275 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-5sqx2"] Oct 30 23:54:09.213802 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:09.213760 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32b07947_679b_40bb_849d_10815894262d.slice/crio-169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64 WatchSource:0}: Error finding container 169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64: Status 404 returned error can't find the container with id 169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64 Oct 30 23:54:09.442707 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.442675 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5sqx2" event={"ID":"32b07947-679b-40bb-849d-10815894262d","Type":"ContainerStarted","Data":"169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64"} Oct 30 23:54:09.444087 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.444064 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vscwx" event={"ID":"58a99b61-5c4d-41c8-9b7a-172aa3edc72b","Type":"ContainerStarted","Data":"7be9d2d9e9a24bdd4480551ebf45c46735e206c3b46f0a600cd04de0f6be7c58"} Oct 30 23:54:09.444238 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.444109 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vscwx" event={"ID":"58a99b61-5c4d-41c8-9b7a-172aa3edc72b","Type":"ContainerStarted","Data":"fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629"} Oct 30 23:54:09.445034 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:09.445012 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dw7gl" event={"ID":"14e77ba7-07ab-4933-9268-760aa9712cb8","Type":"ContainerStarted","Data":"47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9"} Oct 30 23:54:10.139981 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.139949 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:10.140147 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.140018 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:10.140147 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.140036 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:10.143388 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.143364 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:10.143507 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.143387 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:10.144007 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.143990 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:10.144125 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.144047 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:10.144180 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.144131 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:10.144319 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.144300 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:10.144372 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.144361 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:10.449645 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.449611 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vscwx" event={"ID":"58a99b61-5c4d-41c8-9b7a-172aa3edc72b","Type":"ContainerStarted","Data":"fb13dea469b80e81dd34453655d81b9a45ed27ee3753e3cbebc9efcd9a7efa48"} Oct 30 23:54:10.928938 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:10.928853 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7jkq_78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/dns-node-resolver/0.log" Oct 30 23:54:11.455108 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.455017 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dw7gl" event={"ID":"14e77ba7-07ab-4933-9268-760aa9712cb8","Type":"ContainerStarted","Data":"0d52fc76d6c48ed6de426707f8faf0c7433c092dfcdc5138e8e2590435ae5286"} Oct 30 23:54:11.455506 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.455121 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-dw7gl" event={"ID":"14e77ba7-07ab-4933-9268-760aa9712cb8","Type":"ContainerStarted","Data":"6a8b56506b44cca27e8c70630888340b4e813918098c833c1a6ff3863c1a052b"} Oct 30 23:54:11.455506 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.455362 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:11.729420 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.729349 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qqrz_827a07c8-e408-4526-a58b-4f7982a7dcaa/node-ca/0.log" Oct 30 23:54:11.767429 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.767363 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/dns-default-dw7gl" podStartSLOduration=1.852603276 podStartE2EDuration="3.767343479s" totalImagesPullingTime="1.914740203s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:11.475870208 +0000 UTC m=+45.889062292" watchObservedRunningTime="2026-10-30 23:54:11.767343479 +0000 UTC m=+46.180535563" Oct 30 23:54:11.768437 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.768412 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:54:11.771615 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.771596 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.776811 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.776773 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 23:54:11.776811 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.776770 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 23:54:11.776964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.776911 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qtpsm\"" Oct 30 23:54:11.782226 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.782209 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 23:54:11.782400 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.782381 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 23:54:11.782467 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.782435 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 23:54:11.782517 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.782490 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 23:54:11.788385 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.788366 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:54:11.851512 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851480 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851512 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851514 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851745 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851533 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851745 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851592 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851883 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851768 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851883 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851809 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.851985 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.851894 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.876166 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.876125 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 23:54:11.952652 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952610 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952652 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952657 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952931 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952679 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952931 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952931 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952727 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952931 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952809 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.952931 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.952829 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.953151 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.953054 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.953730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.953709 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.954044 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.953993 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.954243 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.954221 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.955834 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.955816 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.955922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.955900 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:11.965087 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:11.965067 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:12.083935 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.083893 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:12.358891 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.358800 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:54:12.363691 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:12.363663 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fbfc93b_8923_4b81_9f55_c4578c1c25f3.slice/crio-e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11 WatchSource:0}: Error finding container e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11: Status 404 returned error can't find the container with id e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11 Oct 30 23:54:12.459484 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.459450 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-vscwx" event={"ID":"58a99b61-5c4d-41c8-9b7a-172aa3edc72b","Type":"ContainerStarted","Data":"a7b42feea7c8648408a46a98a444b5abe69dec03252093e215feecbae99f7717"} Oct 30 23:54:12.460811 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.460766 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6799bcd5d4-r6r54" event={"ID":"3fbfc93b-8923-4b81-9f55-c4578c1c25f3","Type":"ContainerStarted","Data":"e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11"} Oct 30 23:54:12.462201 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.462174 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-5sqx2" event={"ID":"32b07947-679b-40bb-849d-10815894262d","Type":"ContainerStarted","Data":"634e57d14037211b9c9994682d07829d85339bb51a14e92d72b30a250f5d9ef6"} Oct 30 23:54:12.478580 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.478437 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-vscwx" podStartSLOduration=2.155280956 podStartE2EDuration="4.478424848s" totalImagesPullingTime="2.323143892s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:12.477575979 +0000 UTC m=+46.890768082" watchObservedRunningTime="2026-10-30 23:54:12.478424848 +0000 UTC m=+46.891616931" Oct 30 23:54:12.494992 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.494949 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-5sqx2" podStartSLOduration=1.640972085 podStartE2EDuration="4.494923167s" totalImagesPullingTime="2.853951082s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:12.49381618 +0000 UTC m=+46.907008263" watchObservedRunningTime="2026-10-30 23:54:12.494923167 +0000 UTC m=+46.908115250" Oct 30 23:54:12.527300 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:12.527272 2572 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5sqx2_32b07947-679b-40bb-849d-10815894262d/serve-healthcheck-canary/0.log" Oct 30 23:54:14.153894 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.153863 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:54:14.157040 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.157019 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.163339 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.163311 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 23:54:14.170413 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.169856 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:54:14.172658 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172633 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172770 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172692 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172849 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172808 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172849 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172832 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172849 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172866 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172978 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172921 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.172978 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.172973 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274257 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274216 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274309 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274338 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274366 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274393 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274449 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274419 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.274922 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274897 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275014 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.274983 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.275046 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275242 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.275216 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275332 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.275306 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275332 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.275321 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.275837 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.275815 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.277305 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.277285 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.277881 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.277856 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.286986 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.286964 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.470413 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.470380 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:14.755247 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:14.755179 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:54:14.758638 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:14.758597 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod398381a0_d000_45eb_8c18_a1f6b2f2ed60.slice/crio-52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8 WatchSource:0}: Error finding container 52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8: Status 404 returned error can't find the container with id 52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8 Oct 30 23:54:15.473269 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:15.473236 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6799bcd5d4-r6r54" event={"ID":"3fbfc93b-8923-4b81-9f55-c4578c1c25f3","Type":"ContainerStarted","Data":"cc484ead940bff20eeccc55c29b23bbf29b19187d163830003105fddb88f826b"} Oct 30 23:54:15.474565 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:15.474531 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d4f6d76-jd8lq" event={"ID":"398381a0-d000-45eb-8c18-a1f6b2f2ed60","Type":"ContainerStarted","Data":"52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8"} Oct 30 23:54:15.505845 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:15.505761 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-6799bcd5d4-r6r54" podStartSLOduration=1.684794457 podStartE2EDuration="4.505744846s" totalImagesPullingTime="2.820950389s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:15.505001511 +0000 UTC m=+49.918193595" watchObservedRunningTime="2026-10-30 23:54:15.505744846 +0000 UTC m=+49.918936915" Oct 30 23:54:16.481279 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.481238 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d4f6d76-jd8lq" event={"ID":"398381a0-d000-45eb-8c18-a1f6b2f2ed60","Type":"ContainerStarted","Data":"5a2c9c5812d5c887a8f698d60bec19814a03f491478d7b59b7cbaa7844463ed3"} Oct 30 23:54:16.525081 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.525024 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-b4d4f6d76-jd8lq" podStartSLOduration=0.973385413 podStartE2EDuration="2.525007017s" totalImagesPullingTime="1.551621604s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:16.524085792 +0000 UTC m=+50.937277875" watchObservedRunningTime="2026-10-30 23:54:16.525007017 +0000 UTC m=+50.938199100" Oct 30 23:54:16.585545 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.585452 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n"] Oct 30 23:54:16.588801 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.588767 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.597973 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.597951 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:54:16.598063 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.598041 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:54:16.598343 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.598329 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:54:16.600932 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.600918 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:54:16.601336 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.601321 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mjqb7\"" Oct 30 23:54:16.602967 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.602942 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n"] Oct 30 23:54:16.603729 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.603712 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:54:16.633860 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.633831 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7"] Oct 30 23:54:16.641057 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.641025 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.656677 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.656644 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-q2kgm"] Oct 30 23:54:16.659959 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.659939 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7"] Oct 30 23:54:16.660060 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.660046 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695087 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695062 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695220 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695095 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.695220 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695112 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pls9z\" (UniqueName: \"kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.695220 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695131 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.695220 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695151 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695278 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695294 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695312 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695329 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695346 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695399 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695557 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695424 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695557 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695439 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.695557 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.695642 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695603 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695642 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695621 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6mh76\" (UniqueName: \"kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.695642 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695639 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.695756 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695733 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wdt66\" (UniqueName: \"kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.695820 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.695770 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.698516 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.698500 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:54:16.698606 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.698583 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:54:16.708405 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708380 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:54:16.708405 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708392 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:54:16.708549 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708493 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:54:16.708549 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708494 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9slr\"" Oct 30 23:54:16.708620 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708577 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:54:16.708620 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.708582 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fptsc\"" Oct 30 23:54:16.796485 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796446 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796505 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdt66\" (UniqueName: \"kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796525 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796542 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796562 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796577 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pls9z\" (UniqueName: \"kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796604 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.796636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796633 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796682 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796710 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796744 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796772 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796827 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796873 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796897 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.797018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.796985 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797489 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797045 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.797489 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797018 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh76\" (UniqueName: \"kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.797489 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797363 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797489 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797420 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797489 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797463 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797810 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797648 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.797810 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797696 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.797810 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.797710 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.798399 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.798286 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.798636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.798309 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.798636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.798320 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.799571 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.799538 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.799654 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.799633 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.799769 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.799755 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.800179 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.800164 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.800563 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.800540 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.800619 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.800603 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.809526 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.809505 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdt66\" (UniqueName: \"kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.809606 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.809576 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh76\" (UniqueName: \"kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.809711 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.809695 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pls9z\" (UniqueName: \"kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.903864 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.903772 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:16.955183 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.955156 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:16.975559 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:16.975534 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:16.986037 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:16.986009 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd775e8d4_9d4a_4779_a5c5_4442e8d3a306.slice/crio-fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01 WatchSource:0}: Error finding container fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01: Status 404 returned error can't find the container with id fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01 Oct 30 23:54:17.192226 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:17.192202 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9215d31_9f16_4011_a873_2995f3e65cf8.slice/crio-2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5 WatchSource:0}: Error finding container 2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5: Status 404 returned error can't find the container with id 2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5 Oct 30 23:54:17.215316 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.215292 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n"] Oct 30 23:54:17.262424 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.262403 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7"] Oct 30 23:54:17.267091 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:17.267066 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod46826886_bbcb_4f1e_ae2a_94f7feb26d76.slice/crio-7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9 WatchSource:0}: Error finding container 7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9: Status 404 returned error can't find the container with id 7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9 Oct 30 23:54:17.487047 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.486947 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" event={"ID":"46826886-bbcb-4f1e-ae2a-94f7feb26d76","Type":"ContainerStarted","Data":"7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9"} Oct 30 23:54:17.488866 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.488825 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2kgm" event={"ID":"d775e8d4-9d4a-4779-a5c5-4442e8d3a306","Type":"ContainerStarted","Data":"fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01"} Oct 30 23:54:17.491262 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.491238 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" event={"ID":"b9215d31-9f16-4011-a873-2995f3e65cf8","Type":"ContainerStarted","Data":"cc4e3d19159587424e7827cf915e9fbdf1a00fe854c53d09218ce200dc6ed023"} Oct 30 23:54:17.491360 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.491295 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" event={"ID":"b9215d31-9f16-4011-a873-2995f3e65cf8","Type":"ContainerStarted","Data":"dbd1942102c2a57653da0f76648f83a9449b683058fc126c07dc53665a51f1bf"} Oct 30 23:54:17.491360 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.491327 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" event={"ID":"b9215d31-9f16-4011-a873-2995f3e65cf8","Type":"ContainerStarted","Data":"2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5"} Oct 30 23:54:17.648281 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.648247 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:17.652014 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.651991 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.664837 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665282 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665319 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665348 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665362 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665588 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wcgjs\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665733 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.665845 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.666032 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:54:17.667814 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.666055 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:54:17.681353 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.681326 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:17.705423 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705388 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705579 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705490 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705579 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705521 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705590 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705627 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705658 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705885 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705762 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705885 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705836 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.705885 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705864 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.706007 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705932 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.706007 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705975 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.706007 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.705990 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.706141 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.706055 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806543 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806458 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806543 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806512 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806543 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806544 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806574 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806598 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806619 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806654 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806707 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806735 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806764 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.806831 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806825 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.807192 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806877 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.807192 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.806909 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.807192 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.807053 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.807401 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.807376 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.807459 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.807435 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.810033 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.809984 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.810323 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.810306 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.810524 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.810504 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.810696 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.810680 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.813070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.812644 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.813070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.812677 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.813070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.812983 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.813070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.813019 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.813070 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.813049 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.818672 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.818649 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:17.979380 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:17.979356 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:18.277856 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.277532 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:18.498038 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.497999 2572 generic.go:356] "Generic (PLEG): container finished" podID="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" containerID="8d2515a3573576b10c118226216d57c39a9f278f2ea5654313a5787804acc575" exitCode=0 Oct 30 23:54:18.498496 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.498088 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2kgm" event={"ID":"d775e8d4-9d4a-4779-a5c5-4442e8d3a306","Type":"ContainerDied","Data":"8d2515a3573576b10c118226216d57c39a9f278f2ea5654313a5787804acc575"} Oct 30 23:54:18.500636 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.500612 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" event={"ID":"b9215d31-9f16-4011-a873-2995f3e65cf8","Type":"ContainerStarted","Data":"ee991d76eea3f301bcf33a9e9a762c6c3074a7a445674a0f04a27c8a557ca38a"} Oct 30 23:54:18.502905 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.502881 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"f1d9c92f87b221ae6448aa10ffb6a587a2be53b5e62833d4354e231cea180ec9"} Oct 30 23:54:18.537290 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:18.537228 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" podStartSLOduration=1.481121554 podStartE2EDuration="2.537208135s" totalImagesPullingTime="1.056086581s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:18.536244431 +0000 UTC m=+52.949436516" watchObservedRunningTime="2026-10-30 23:54:18.537208135 +0000 UTC m=+52.950400225" Oct 30 23:54:19.512482 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.512448 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" event={"ID":"46826886-bbcb-4f1e-ae2a-94f7feb26d76","Type":"ContainerStarted","Data":"b41a3ae02824250fb7500c8ffc2beff0d287c5d88f31836be7ab86cd4199e635"} Oct 30 23:54:19.512880 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.512543 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" event={"ID":"46826886-bbcb-4f1e-ae2a-94f7feb26d76","Type":"ContainerStarted","Data":"99ee82bddcfad51cbcd37b03906930fe896e762e4be8c5beba2612b849e5ab54"} Oct 30 23:54:19.512880 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.512573 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" event={"ID":"46826886-bbcb-4f1e-ae2a-94f7feb26d76","Type":"ContainerStarted","Data":"070b65e92d60c6d6989839c4da8c7acc6321b4e43a87292601fe50eedb4bd159"} Oct 30 23:54:19.514746 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.514654 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2kgm" event={"ID":"d775e8d4-9d4a-4779-a5c5-4442e8d3a306","Type":"ContainerStarted","Data":"ec9a9e6899b072694af3a8327c2b173178e43b69fc22f85971519ba2299fdfab"} Oct 30 23:54:19.514746 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.514716 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-q2kgm" event={"ID":"d775e8d4-9d4a-4779-a5c5-4442e8d3a306","Type":"ContainerStarted","Data":"94559f26ff20cc851576bad183be494971e63acaa98581756d6b1345602e0910"} Oct 30 23:54:19.532553 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.532508 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" podStartSLOduration=1.9108503369999998 podStartE2EDuration="3.532492273s" totalImagesPullingTime="1.621641936s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:19.531245275 +0000 UTC m=+53.944437358" watchObservedRunningTime="2026-10-30 23:54:19.532492273 +0000 UTC m=+53.945684356" Oct 30 23:54:19.551474 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:19.551433 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-q2kgm" podStartSLOduration=2.511004818 podStartE2EDuration="3.551419853s" totalImagesPullingTime="948.643453ms" totalInitContainerRuntime="91.771582ms" isStatefulPod=true podCreationTimestamp="2026-10-30 23:54:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:19.549999982 +0000 UTC m=+53.963192065" watchObservedRunningTime="2026-10-30 23:54:19.551419853 +0000 UTC m=+53.964611935" Oct 30 23:54:20.519508 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:20.519478 2572 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" exitCode=0 Oct 30 23:54:20.519924 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:20.519569 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} Oct 30 23:54:21.464598 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:21.464570 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:22.085443 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.084628 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:22.085443 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.084741 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:22.090555 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.090531 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:22.419901 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.419876 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:22.528595 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.528560 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} Oct 30 23:54:22.528706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.528645 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} Oct 30 23:54:22.528706 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.528689 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} Oct 30 23:54:22.532437 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:22.532416 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:23.535011 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.534965 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} Oct 30 23:54:23.535363 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.535034 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} Oct 30 23:54:23.535363 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.535069 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerStarted","Data":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} Oct 30 23:54:23.571503 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.571448 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.811356668 podStartE2EDuration="6.571429891s" totalImagesPullingTime="3.745578813s" totalInitContainerRuntime="14.49441ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:17 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:23.570220652 +0000 UTC m=+57.983412735" watchObservedRunningTime="2026-10-30 23:54:23.571429891 +0000 UTC m=+57.984621974" Oct 30 23:54:23.766119 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.766083 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:54:23.821025 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.820993 2572 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:54:23.824248 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.824229 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.836407 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.836386 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:54:23.866549 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866518 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866664 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866557 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866712 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866678 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866712 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866701 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866821 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866795 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866918 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866869 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866918 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866892 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.866985 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.866944 2572 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967718 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967751 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967755 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967811 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967851 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967872 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967905 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.967964 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.967931 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.968185 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.968164 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.968795 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.968748 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.969018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.968874 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.969018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.968921 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.969018 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.969015 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.970574 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.970550 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.970731 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.970713 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:23.992509 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:23.992488 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:24.134762 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.134660 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:24.413392 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.413325 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:54:24.416990 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:24.416964 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3027d685_5695_4d70_8759_5b28d323af4f.slice/crio-a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f WatchSource:0}: Error finding container a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f: Status 404 returned error can't find the container with id a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f Oct 30 23:54:24.470826 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.470803 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:24.540345 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.540311 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d4889bc-vs299" event={"ID":"3027d685-5695-4d70-8759-5b28d323af4f","Type":"ContainerStarted","Data":"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7"} Oct 30 23:54:24.540696 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.540364 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d4889bc-vs299" event={"ID":"3027d685-5695-4d70-8759-5b28d323af4f","Type":"ContainerStarted","Data":"a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f"} Oct 30 23:54:24.560298 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:24.560046 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-588d4889bc-vs299" podStartSLOduration=1.560030041 podStartE2EDuration="1.560030041s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:24.559032487 +0000 UTC m=+58.972224579" watchObservedRunningTime="2026-10-30 23:54:24.560030041 +0000 UTC m=+58.973222127" Oct 30 23:54:31.942693 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.942646 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:31.942693 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.942702 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:31.943286 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.942834 2572 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:31.945362 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.945345 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:31.945362 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.945347 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:31.945697 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.945683 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:31.955810 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.955772 2572 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:31.956089 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.956071 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:31.956215 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.956196 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:31.966354 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:31.966330 2572 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:32.056641 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.056608 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:32.064730 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.064708 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:32.065583 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.065565 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:32.074664 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.074537 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:32.076213 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.076189 2572 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:32.082601 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.082577 2572 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:32.347766 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.347685 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-8nvcl"] Oct 30 23:54:32.350748 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:32.350720 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa89d9ae_29cc_4664_ae9d_ca28b785c7bd.slice/crio-6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4 WatchSource:0}: Error finding container 6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4: Status 404 returned error can't find the container with id 6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4 Oct 30 23:54:32.372041 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.372013 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-45ssd"] Oct 30 23:54:32.377098 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:32.377075 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4285655f_2bda_4d2b_87f5_b9506c3fdb51.slice/crio-52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3 WatchSource:0}: Error finding container 52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3: Status 404 returned error can't find the container with id 52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3 Oct 30 23:54:32.392602 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.392576 2572 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-6dpdv"] Oct 30 23:54:32.397120 ip-10-0-130-152 kubenswrapper[2572]: W1030 23:54:32.397093 2572 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d1f10bd_b2d0_486d_9caa_21b650348f93.slice/crio-4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981 WatchSource:0}: Error finding container 4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981: Status 404 returned error can't find the container with id 4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981 Oct 30 23:54:32.569980 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.569941 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-45ssd" event={"ID":"4285655f-2bda-4d2b-87f5-b9506c3fdb51","Type":"ContainerStarted","Data":"52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3"} Oct 30 23:54:32.571059 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.571031 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dpdv" event={"ID":"4d1f10bd-b2d0-486d-9caa-21b650348f93","Type":"ContainerStarted","Data":"4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981"} Oct 30 23:54:32.572140 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:32.572123 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nvcl" event={"ID":"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd","Type":"ContainerStarted","Data":"6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4"} Oct 30 23:54:34.135481 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.135441 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:34.135988 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.135551 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:34.144146 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.144119 2572 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:34.586367 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.586332 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dpdv" event={"ID":"4d1f10bd-b2d0-486d-9caa-21b650348f93","Type":"ContainerStarted","Data":"1a4bde02e6ad0d1bdcbc4aacbe4dea77a0037da8170f835b2b100965c7a3b43b"} Oct 30 23:54:34.586557 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.586388 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-6dpdv" event={"ID":"4d1f10bd-b2d0-486d-9caa-21b650348f93","Type":"ContainerStarted","Data":"52ac009201fd001ba82ac1f2ea5fabc48f7ac5ba22ea052d13a1acf0532bb737"} Oct 30 23:54:34.596693 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.596657 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:34.605514 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.605111 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-6dpdv" podStartSLOduration=67.417100556 podStartE2EDuration="1m8.605095547s" totalImagesPullingTime="1.187994991s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:34.603892678 +0000 UTC m=+69.017084762" watchObservedRunningTime="2026-10-30 23:54:34.605095547 +0000 UTC m=+69.018287631" Oct 30 23:54:34.660241 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.660204 2572 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:54:34.832477 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:34.832422 2572 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-dbh42_5bdcd7f1-7d32-40eb-8daf-8c83e77e8296/konnectivity-agent/0.log" limit=1024 Oct 30 23:54:36.596880 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:36.596838 2572 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-8nvcl" event={"ID":"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd","Type":"ContainerStarted","Data":"2652bee8f5a29dfc392cbc8b777b18cba7e4be469e72a0c91764a4f3c8b03940"} Oct 30 23:54:36.597285 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:36.596978 2572 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:36.614417 ip-10-0-130-152 kubenswrapper[2572]: I1030 23:54:36.614359 2572 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-8nvcl" podStartSLOduration=67.370787057 podStartE2EDuration="1m10.614343827s" totalImagesPullingTime="3.24355677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:53:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:36.613034691 +0000 UTC m=+71.026226778" watchObservedRunningTime="2026-10-30 23:54:36.614343827 +0000 UTC m=+71.027535910" Oct 30 23:54:37.906100 ip-10-0-130-152 systemd[1]: Stopping Kubernetes Kubelet... Oct 30 23:54:37.920197 ip-10-0-130-152 systemd[1]: kubelet.service: Deactivated successfully. Oct 30 23:54:37.920364 ip-10-0-130-152 systemd[1]: Stopped Kubernetes Kubelet. Oct 30 23:54:37.920411 ip-10-0-130-152 systemd[1]: kubelet.service: Consumed 4.170s CPU time. Oct 30 23:54:37.933569 ip-10-0-130-152 systemd[1]: Starting Kubernetes Kubelet... Oct 30 23:54:37.996472 ip-10-0-130-152 kubenswrapper[6234]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:37.996472 ip-10-0-130-152 kubenswrapper[6234]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version. Oct 30 23:54:37.996472 ip-10-0-130-152 kubenswrapper[6234]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:37.996472 ip-10-0-130-152 kubenswrapper[6234]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information. Oct 30 23:54:37.998543 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998465 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:37.998543 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998544 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998548 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998551 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998554 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998556 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998559 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998562 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998569 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998572 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998575 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998578 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998581 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998583 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998586 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998588 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998591 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998593 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998596 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998599 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:37.998616 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998601 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998604 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998606 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998611 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998615 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998618 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998621 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998624 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998627 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998630 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998632 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998635 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998637 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998640 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998642 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998645 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998647 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998650 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998653 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998656 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:37.999112 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998658 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998661 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998663 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998666 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998668 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998671 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998675 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998679 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998682 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998685 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998688 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998690 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998693 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998695 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998698 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998700 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998703 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998706 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998709 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:37.999599 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998711 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998714 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998717 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998719 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998722 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998725 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998728 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998730 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998733 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998735 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998737 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998740 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998743 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998745 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998748 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998751 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998753 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998756 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998758 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.000124 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998761 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998764 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998766 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998768 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998771 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998773 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998776 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998797 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998802 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998806 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998811 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998816 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998819 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998821 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998824 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998827 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998829 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998832 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998835 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.000584 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998838 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998840 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998843 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998845 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998848 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998851 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998854 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998856 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.998859 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999618 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999624 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999627 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999630 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999632 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999636 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999639 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999642 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999644 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999647 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999650 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.001076 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999652 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999655 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999658 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999661 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999664 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999667 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999669 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999672 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999675 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999678 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999680 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999683 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999686 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999688 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999691 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999694 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999696 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999699 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999701 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.001596 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999704 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999706 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999710 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999713 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999715 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999718 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999720 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999723 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999726 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999728 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999731 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999733 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999736 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999739 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999742 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999744 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999747 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999750 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999752 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999755 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.002085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999757 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999760 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999764 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999768 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999772 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999776 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999795 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999799 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999803 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999809 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999814 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999817 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999820 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999822 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999826 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999829 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999832 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999835 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999837 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.002605 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999840 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999843 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999846 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999848 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999851 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999853 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999858 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999863 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999866 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999868 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999871 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999873 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999876 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999878 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999881 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999883 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999886 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999888 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999891 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.003095 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999893 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999896 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999898 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999901 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999903 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999906 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999908 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999910 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999913 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999915 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999919 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999921 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999924 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999927 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999929 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999932 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999935 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:37.999942 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000473 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000478 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.003567 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000481 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000484 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000487 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000489 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000492 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000494 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000497 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000499 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000502 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000505 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000509 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000512 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000516 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000519 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000521 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000524 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000527 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000530 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000532 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000535 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.004088 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000537 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000540 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000550 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000553 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000555 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000558 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000561 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000563 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000566 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000568 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000571 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000573 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000575 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000578 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000581 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000583 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000586 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000589 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000591 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000594 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.004607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000597 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000599 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000602 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000605 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000607 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000610 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000612 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000615 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000617 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000620 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000622 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000625 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000628 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000630 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000633 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000641 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000643 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000647 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000649 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000651 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.005128 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000654 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000656 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000659 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000661 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000664 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000666 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000669 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000671 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000673 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000676 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000678 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000681 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000683 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000686 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000688 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000692 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000696 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000699 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.005771 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000702 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000705 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000707 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000710 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000712 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000715 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000718 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000720 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000723 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000725 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000733 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000736 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000739 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000741 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000744 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000746 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000748 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000751 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000753 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000756 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.006455 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000759 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000762 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000764 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000766 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000769 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.000771 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000890 6234 flags.go:64] FLAG: --address="0.0.0.0" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000901 6234 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000914 6234 flags.go:64] FLAG: --anonymous-auth="true" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000919 6234 flags.go:64] FLAG: --application-metrics-count-limit="100" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000924 6234 flags.go:64] FLAG: --authentication-token-webhook="false" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000927 6234 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000931 6234 flags.go:64] FLAG: --authorization-mode="AlwaysAllow" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000935 6234 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000938 6234 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000941 6234 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000949 6234 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000952 6234 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000956 6234 flags.go:64] FLAG: --cgroup-driver="cgroupfs" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000958 6234 flags.go:64] FLAG: --cgroup-root="" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000961 6234 flags.go:64] FLAG: --cgroups-per-qos="true" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000964 6234 flags.go:64] FLAG: --client-ca-file="" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000967 6234 flags.go:64] FLAG: --cloud-provider="external" Oct 30 23:54:38.006981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000980 6234 flags.go:64] FLAG: --cluster-dns="[]" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000987 6234 flags.go:64] FLAG: --cluster-domain="" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000990 6234 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000993 6234 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000996 6234 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.000999 6234 flags.go:64] FLAG: --container-log-max-files="5" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001004 6234 flags.go:64] FLAG: --container-log-max-size="10Mi" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001007 6234 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001010 6234 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001013 6234 flags.go:64] FLAG: --containerd-namespace="k8s.io" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001016 6234 flags.go:64] FLAG: --contention-profiling="false" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001019 6234 flags.go:64] FLAG: --cpu-cfs-quota="true" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001025 6234 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001028 6234 flags.go:64] FLAG: --cpu-manager-policy="none" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001031 6234 flags.go:64] FLAG: --cpu-manager-policy-options="" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001035 6234 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001038 6234 flags.go:64] FLAG: --enable-controller-attach-detach="true" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001041 6234 flags.go:64] FLAG: --enable-debugging-handlers="true" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001044 6234 flags.go:64] FLAG: --enable-load-reader="false" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001046 6234 flags.go:64] FLAG: --enable-server="true" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001049 6234 flags.go:64] FLAG: --enforce-node-allocatable="[pods]" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001054 6234 flags.go:64] FLAG: --event-burst="100" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001058 6234 flags.go:64] FLAG: --event-qps="50" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001061 6234 flags.go:64] FLAG: --event-storage-age-limit="default=0" Oct 30 23:54:38.007539 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001063 6234 flags.go:64] FLAG: --event-storage-event-limit="default=0" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001068 6234 flags.go:64] FLAG: --eviction-hard="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001072 6234 flags.go:64] FLAG: --eviction-max-pod-grace-period="0" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001075 6234 flags.go:64] FLAG: --eviction-minimum-reclaim="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001079 6234 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001082 6234 flags.go:64] FLAG: --eviction-soft="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001085 6234 flags.go:64] FLAG: --eviction-soft-grace-period="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001088 6234 flags.go:64] FLAG: --exit-on-lock-contention="false" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001091 6234 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001095 6234 flags.go:64] FLAG: --experimental-mounter-path="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001098 6234 flags.go:64] FLAG: --fail-cgroupv1="true" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001101 6234 flags.go:64] FLAG: --fail-swap-on="true" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001103 6234 flags.go:64] FLAG: --feature-gates="" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001110 6234 flags.go:64] FLAG: --file-check-frequency="20s" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001113 6234 flags.go:64] FLAG: --global-housekeeping-interval="1m0s" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001116 6234 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001119 6234 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001122 6234 flags.go:64] FLAG: --healthz-port="10248" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001125 6234 flags.go:64] FLAG: --help="false" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001128 6234 flags.go:64] FLAG: --hostname-override="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001133 6234 flags.go:64] FLAG: --housekeeping-interval="10s" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001136 6234 flags.go:64] FLAG: --http-check-frequency="20s" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001138 6234 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins" Oct 30 23:54:38.008169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001143 6234 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001146 6234 flags.go:64] FLAG: --image-gc-high-threshold="85" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001149 6234 flags.go:64] FLAG: --image-gc-low-threshold="80" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001151 6234 flags.go:64] FLAG: --image-service-endpoint="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001154 6234 flags.go:64] FLAG: --kernel-memcg-notification="false" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001157 6234 flags.go:64] FLAG: --kube-api-burst="100" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001160 6234 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001163 6234 flags.go:64] FLAG: --kube-api-qps="50" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001166 6234 flags.go:64] FLAG: --kube-reserved="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001169 6234 flags.go:64] FLAG: --kube-reserved-cgroup="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001173 6234 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001176 6234 flags.go:64] FLAG: --kubelet-cgroups="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001178 6234 flags.go:64] FLAG: --local-storage-capacity-isolation="true" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001181 6234 flags.go:64] FLAG: --lock-file="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001184 6234 flags.go:64] FLAG: --log-cadvisor-usage="false" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001187 6234 flags.go:64] FLAG: --log-flush-frequency="5s" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001190 6234 flags.go:64] FLAG: --log-json-info-buffer-size="0" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001195 6234 flags.go:64] FLAG: --log-json-split-stream="false" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001198 6234 flags.go:64] FLAG: --log-text-info-buffer-size="0" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001202 6234 flags.go:64] FLAG: --log-text-split-stream="false" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001205 6234 flags.go:64] FLAG: --logging-format="text" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001208 6234 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001212 6234 flags.go:64] FLAG: --make-iptables-util-chains="true" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001214 6234 flags.go:64] FLAG: --manifest-url="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001217 6234 flags.go:64] FLAG: --manifest-url-header="" Oct 30 23:54:38.008736 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001222 6234 flags.go:64] FLAG: --max-housekeeping-interval="15s" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001225 6234 flags.go:64] FLAG: --max-open-files="1000000" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001228 6234 flags.go:64] FLAG: --max-pods="110" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001231 6234 flags.go:64] FLAG: --maximum-dead-containers="-1" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001235 6234 flags.go:64] FLAG: --maximum-dead-containers-per-container="1" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001238 6234 flags.go:64] FLAG: --memory-manager-policy="None" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001241 6234 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001244 6234 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001247 6234 flags.go:64] FLAG: --node-ip="0.0.0.0" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001249 6234 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001260 6234 flags.go:64] FLAG: --node-status-max-images="50" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001263 6234 flags.go:64] FLAG: --node-status-update-frequency="10s" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001266 6234 flags.go:64] FLAG: --oom-score-adj="-999" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001270 6234 flags.go:64] FLAG: --pod-cidr="" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001272 6234 flags.go:64] FLAG: --pod-manifest-path="" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001275 6234 flags.go:64] FLAG: --pod-max-pids="-1" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001278 6234 flags.go:64] FLAG: --pods-per-core="0" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001282 6234 flags.go:64] FLAG: --port="10250" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001285 6234 flags.go:64] FLAG: --protect-kernel-defaults="false" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001288 6234 flags.go:64] FLAG: --provider-id="aws:///us-east-2a/i-0100f67647ce5dcca" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001291 6234 flags.go:64] FLAG: --qos-reserved="" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001294 6234 flags.go:64] FLAG: --read-only-port="10255" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001297 6234 flags.go:64] FLAG: --register-node="true" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001300 6234 flags.go:64] FLAG: --register-with-taints="" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001304 6234 flags.go:64] FLAG: --registry-burst="10" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001306 6234 flags.go:64] FLAG: --registry-qps="5" Oct 30 23:54:38.009361 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001309 6234 flags.go:64] FLAG: --reserved-cpus="" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001317 6234 flags.go:64] FLAG: --reserved-memory="" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001321 6234 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001324 6234 flags.go:64] FLAG: --root-dir="/var/lib/kubelet" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001326 6234 flags.go:64] FLAG: --rotate-certificates="false" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001329 6234 flags.go:64] FLAG: --rotate-server-certificates="false" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001332 6234 flags.go:64] FLAG: --runonce="false" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001335 6234 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001338 6234 flags.go:64] FLAG: --runtime-request-timeout="2m0s" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001341 6234 flags.go:64] FLAG: --seccomp-default="false" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001345 6234 flags.go:64] FLAG: --serialize-image-pulls="true" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001347 6234 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001351 6234 flags.go:64] FLAG: --storage-driver-db="cadvisor" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001354 6234 flags.go:64] FLAG: --storage-driver-host="localhost:8086" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001357 6234 flags.go:64] FLAG: --storage-driver-password="root" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001360 6234 flags.go:64] FLAG: --storage-driver-secure="false" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001362 6234 flags.go:64] FLAG: --storage-driver-table="stats" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001365 6234 flags.go:64] FLAG: --storage-driver-user="root" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001368 6234 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001371 6234 flags.go:64] FLAG: --sync-frequency="1m0s" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001374 6234 flags.go:64] FLAG: --system-cgroups="" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001377 6234 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001382 6234 flags.go:64] FLAG: --system-reserved-cgroup="" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001386 6234 flags.go:64] FLAG: --tls-cert-file="" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001389 6234 flags.go:64] FLAG: --tls-cipher-suites="[]" Oct 30 23:54:38.010007 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001392 6234 flags.go:64] FLAG: --tls-min-version="" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001395 6234 flags.go:64] FLAG: --tls-private-key-file="" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001398 6234 flags.go:64] FLAG: --topology-manager-policy="none" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001401 6234 flags.go:64] FLAG: --topology-manager-policy-options="" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001403 6234 flags.go:64] FLAG: --topology-manager-scope="container" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001406 6234 flags.go:64] FLAG: --v="2" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001411 6234 flags.go:64] FLAG: --version="false" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001415 6234 flags.go:64] FLAG: --vmodule="" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001419 6234 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001422 6234 flags.go:64] FLAG: --volume-stats-agg-period="1m0s" Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001594 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001599 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001603 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001606 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001609 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001611 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001614 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001618 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001620 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001623 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001626 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001628 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.010591 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001630 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001633 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001636 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001638 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001641 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001643 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001646 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001654 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001658 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001661 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001664 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001667 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001669 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001672 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001674 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001677 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001680 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001682 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001685 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001687 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.011150 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001695 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001698 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001701 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001703 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001706 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001708 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001712 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001718 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001721 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001724 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001727 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001730 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001732 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001735 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001737 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001740 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001742 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001745 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001748 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.011642 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001752 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001756 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001759 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001762 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001764 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001767 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001769 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001772 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001774 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001777 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001800 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001803 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001806 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001808 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001817 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001820 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001823 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001825 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001827 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001830 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.012115 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001833 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001836 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001838 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001841 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001843 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001846 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001849 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001851 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001854 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001856 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001859 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001862 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001864 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001867 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001870 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001872 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001875 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001877 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001880 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.012607 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001883 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001885 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001888 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001890 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001893 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001895 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001897 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001900 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001908 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001911 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001913 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001916 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001918 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001922 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001924 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001927 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.013302 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.001953 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.001999 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002005 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002008 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002011 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002014 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002017 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002020 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002022 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002025 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002027 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002032 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002034 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002037 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002039 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002042 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002045 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002047 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002049 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.013773 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002052 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002054 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002057 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002059 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002062 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002064 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002074 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002077 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002079 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002082 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002086 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002089 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002092 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002094 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002097 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002099 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002102 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002104 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002107 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.014416 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002109 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002112 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002114 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002117 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002119 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002123 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002125 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002127 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002130 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002132 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002135 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002137 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002140 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002142 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002145 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002147 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002149 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002152 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002154 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.014912 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002156 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002164 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002167 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002170 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002173 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002176 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002178 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002181 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002183 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002186 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002189 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002191 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002194 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002196 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002198 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002201 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002203 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002206 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002209 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002212 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002214 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.015398 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002217 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002220 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002222 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002225 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002227 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002229 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002232 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002235 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002237 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002239 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002242 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002244 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002247 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002254 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002257 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002261 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002264 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002268 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002271 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.016036 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002274 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002278 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002280 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002283 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002286 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002288 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002290 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002293 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002296 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.002299 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.007717 6234 server.go:525] "Kubelet version" kubeletVersion="v1.35.9" Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.007733 6234 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK="" Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.007760 6234 watchdog_linux.go:95] "Systemd watchdog is not enabled" Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.007765 6234 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started." Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007846 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007852 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007856 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007860 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007864 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.016585 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007867 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007871 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007875 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007878 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007881 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007884 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007887 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007890 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007892 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007895 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007898 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007901 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007904 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007906 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007909 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007912 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007914 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007917 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007919 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007922 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.017100 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007925 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007927 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007931 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007935 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007938 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007942 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007945 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007948 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007951 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007954 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007957 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007959 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007962 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007965 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007967 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007970 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007972 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007975 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007977 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.017579 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007980 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007982 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007985 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007988 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007990 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007993 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007995 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.007998 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008000 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008003 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008005 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008008 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008010 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008013 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008015 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008018 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008020 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008023 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008025 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.018060 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008028 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008031 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008034 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008037 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008039 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008042 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008044 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008047 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008050 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008052 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008055 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008058 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008060 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008063 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008065 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008068 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008071 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008074 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008076 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.018503 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008079 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008082 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008084 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008086 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008089 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008092 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008094 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008097 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008099 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008102 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008104 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008107 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008109 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008112 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008114 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008117 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008120 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008122 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008125 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008127 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.018985 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008130 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008133 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008136 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008138 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.008166 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008305 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008309 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008312 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008315 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008318 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008322 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008325 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008328 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008331 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008334 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008337 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.019490 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008339 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008342 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008344 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008347 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008349 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008352 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008354 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008357 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008359 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008362 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008364 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008367 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008370 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008372 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008375 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008377 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008380 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008382 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008385 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.019906 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008387 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008390 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008392 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008395 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008397 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008400 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008402 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008405 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008408 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008411 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008413 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008416 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008418 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008420 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008423 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008425 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008428 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008432 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008436 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.020365 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008439 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008442 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008445 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008447 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008450 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008453 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008455 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008458 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008460 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008465 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008468 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008471 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008473 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008475 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008479 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008483 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008486 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008489 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008491 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008494 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.020857 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008497 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008499 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008502 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008505 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008507 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008509 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008512 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008514 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008517 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008519 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008522 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008524 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008527 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008529 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008532 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008534 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008537 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008539 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008542 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.021336 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008544 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008547 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008551 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008553 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008556 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008558 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008561 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008563 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008565 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008568 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008570 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008573 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008575 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008578 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008580 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008583 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008586 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.021845 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008589 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.008613 6234 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]} Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008662 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008667 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008670 6234 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008673 6234 feature_gate.go:386] unrecognized feature gate: Example Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008676 6234 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008678 6234 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008681 6234 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008683 6234 feature_gate.go:386] unrecognized feature gate: UpgradeStatus Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008686 6234 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008690 6234 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release. Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008694 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryption Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008696 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008699 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification Oct 30 23:54:38.022254 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008702 6234 feature_gate.go:386] unrecognized feature gate: NetworkConnect Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008704 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008708 6234 feature_gate.go:386] unrecognized feature gate: Example2 Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008711 6234 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008713 6234 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008716 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008719 6234 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008721 6234 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008723 6234 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008726 6234 feature_gate.go:386] unrecognized feature gate: OVNObservability Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008729 6234 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008731 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2 Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008733 6234 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008736 6234 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008739 6234 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008742 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008744 6234 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008748 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDC Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008751 6234 feature_gate.go:386] unrecognized feature gate: SignatureStores Oct 30 23:54:38.023853 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008754 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008756 6234 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008759 6234 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008761 6234 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008764 6234 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008766 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008769 6234 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008771 6234 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008774 6234 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008776 6234 feature_gate.go:386] unrecognized feature gate: TLSAdherence Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008802 6234 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008807 6234 feature_gate.go:386] unrecognized feature gate: NewOLM Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008811 6234 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008815 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008819 6234 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008826 6234 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008828 6234 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008832 6234 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008834 6234 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall Oct 30 23:54:38.024313 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008836 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008840 6234 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008842 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008845 6234 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008847 6234 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008850 6234 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008852 6234 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008856 6234 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008858 6234 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008861 6234 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008864 6234 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008866 6234 feature_gate.go:386] unrecognized feature gate: DNSNameResolver Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008869 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008872 6234 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008874 6234 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008877 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008879 6234 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008882 6234 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008884 6234 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards Oct 30 23:54:38.024767 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008886 6234 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008889 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008891 6234 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008894 6234 feature_gate.go:386] unrecognized feature gate: ShortCertRotation Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008896 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008898 6234 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008901 6234 feature_gate.go:386] unrecognized feature gate: EventTTL Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008903 6234 feature_gate.go:386] unrecognized feature gate: DualReplica Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008906 6234 feature_gate.go:386] unrecognized feature gate: NoOverlayMode Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008909 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008912 6234 feature_gate.go:386] unrecognized feature gate: EVPN Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008914 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008917 6234 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008919 6234 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008922 6234 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008924 6234 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008927 6234 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008929 6234 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008932 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008934 6234 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup Oct 30 23:54:38.025319 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008937 6234 feature_gate.go:386] unrecognized feature gate: InsightsConfig Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008939 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008942 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008944 6234 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008946 6234 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008949 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008952 6234 feature_gate.go:386] unrecognized feature gate: OSStreams Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008955 6234 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008958 6234 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008961 6234 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release. Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008965 6234 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008970 6234 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008973 6234 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008975 6234 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008978 6234 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:38.008980 6234 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.009040 6234 server.go:957] "Client rotation is on, will bootstrap in background" Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010012 6234 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary" Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010061 6234 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem" Oct 30 23:54:38.025833 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010266 6234 server.go:1014] "Starting client certificate rotation" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010407 6234 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010447 6234 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-10-29 23:48:26 +0000 UTC" deadline="2028-07-02 21:50:09.426543022 +0000 UTC" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010513 6234 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14661h55m31.416038294s" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.010746 6234 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.011934 6234 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.013558 6234 log.go:25] "Validated CRI v1 runtime API" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.015470 6234 log.go:25] "Validated CRI v1 image API" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.016274 6234 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd" Oct 30 23:54:38.026294 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.018297 6234 fs.go:135] Filesystem UUIDs: map[40960bc2-2bb9-4f7c-8263-0e6f0e0e6489:/dev/nvme0n1p4 6beb498b-e7f2-427a-9b79-61dc40a3e89b:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2] Oct 30 23:54:38.026768 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.018317 6234 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64/userdata/shm major:0 minor:209 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2d8e482a7489e7c1d8bb25e103ce9c5a4d63006df4754b09107af3a7e6adb454/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2d8e482a7489e7c1d8bb25e103ce9c5a4d63006df4754b09107af3a7e6adb454/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5/userdata/shm major:0 minor:285 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981/userdata/shm major:0 minor:404 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9/userdata/shm major:0 minor:205 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665/userdata/shm major:0 minor:94 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3/userdata/shm major:0 minor:402 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8/userdata/shm major:0 minor:260 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4/userdata/shm major:0 minor:400 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f/userdata/shm major:0 minor:82 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9/userdata/shm major:0 minor:287 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f/userdata/shm major:0 minor:387 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa/userdata/shm major:0 minor:97 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c97249d546434a48bb4612f4f2e8ba46b234fd76e9e7f7a22671035cc42a440d/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c97249d546434a48bb4612f4f2e8ba46b234fd76e9e7f7a22671035cc42a440d/userdata/shm major:0 minor:75 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10/userdata/shm major:0 minor:86 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11/userdata/shm major:0 minor:249 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c/userdata/shm major:0 minor:90 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f1d9c92f87b221ae6448aa10ffb6a587a2be53b5e62833d4354e231cea180ec9/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f1d9c92f87b221ae6448aa10ffb6a587a2be53b5e62833d4354e231cea180ec9/userdata/shm major:0 minor:324 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629/userdata/shm major:0 minor:207 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01/userdata/shm major:0 minor:289 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~projected/kube-api-access-g25jl:{mountpoint:/var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~projected/kube-api-access-g25jl major:0 minor:203 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~secret/metrics-tls:{mountpoint:/var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~secret/metrics-tls major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~projected/kube-api-access-hgw25:{mountpoint:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~projected/kube-api-access-hgw25 major:0 minor:386 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:381 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:385 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~projected/kube-api-access-jd7df:{mountpoint:/var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~projected/kube-api-access-jd7df major:0 minor:202 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~projected/kube-api-access-wqd5r:{mountpoint:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~projected/kube-api-access-wqd5r major:0 minor:259 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:258 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:257 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~projected/kube-api-access-f6f7l:{mountpoint:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~projected/kube-api-access-f6f7l major:0 minor:246 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-oauth-config:{mountpoint:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-oauth-config major:0 minor:245 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-serving-cert:{mountpoint:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-serving-cert major:0 minor:241 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4285655f-2bda-4d2b-87f5-b9506c3fdb51/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/4285655f-2bda-4d2b-87f5-b9506c3fdb51/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:393 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/44459cea-2fa8-44fb-ad9f-9aeafcdae6df/volumes/kubernetes.io~projected/kube-api-access-9nd8n:{mountpoint:/var/lib/kubelet/pods/44459cea-2fa8-44fb-ad9f-9aeafcdae6df/volumes/kubernetes.io~projected/kube-api-access-9nd8n major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~projected/kube-api-access-pls9z:{mountpoint:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~projected/kube-api-access-pls9z major:0 minor:283 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config major:0 minor:272 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-tls major:0 minor:279 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~projected/kube-api-access-cb42l:{mountpoint:/var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~projected/kube-api-access-cb42l major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~secret/metrics-certs:{mountpoint:/var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~secret/metrics-certs major:0 minor:398 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~projected/kube-api-access-h8cgn:{mountpoint:/var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~projected/kube-api-access-h8cgn major:0 minor:204 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls:{mountpoint:/var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls major:0 minor:197 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/721668b5-4594-43bb-82ae-f8fd5e80fd2e/volumes/kubernetes.io~projected/kube-api-access-rzcpr:{mountpoint:/var/lib/kubelet/pods/721668b5-4594-43bb-82ae-f8fd5e80fd2e/volumes/kubernetes.io~projected/kube-api-access-rzcpr major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/volumes/kubernetes.io~projected/kube-api-access-9b787:{mountpoint:/var/lib/kubelet/pods/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/volumes/kubernetes.io~projected/kube-api-access-9b787 major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/827a07c8-e408-4526-a58b-4f7982a7dcaa/volumes/kubernetes.io~projected/kube-api-access-8z54h:{mountpoint:/var/lib/kubelet/pods/827a07c8-e408-4526-a58b-4f7982a7dcaa/volumes/kubernetes.io~projected/kube-api-access-8z54h major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8411675f-2736-483b-8a74-a15c12287ea1/volumes/kubernetes.io~projected/kube-api-access-7ctp6:{mountpoint:/var/lib/kubelet/pods/8411675f-2736-483b-8a74-a15c12287ea1/volumes/kubernetes.io~projected/kube-api-access-7ctp6 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/cluster-tls-config/alertmanager/10:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/cluster-tls-config/alertmanager/10 major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/alertmanager/9:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/alertmanager/9 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/init-config-reloader/6:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/init-config-reloader/6 major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~empty-dir/config-out:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~empty-dir/config-out major:0 minor:311 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/kube-api-access-lsh47:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/kube-api-access-lsh47 major:0 minor:317 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/tls-assets:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/tls-assets major:0 minor:315 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/cluster-tls-config:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/cluster-tls-config major:0 minor:305 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/config-volume:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/config-volume major:0 minor:312 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy major:0 minor:310 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric major:0 minor:309 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web major:0 minor:316 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-main-tls:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-main-tls major:0 minor:313 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/web-config:{mountpoint:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/web-config major:0 minor:314 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/tmp major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~projected/kube-api-access-58qmb:{mountpoint:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~projected/kube-api-access-58qmb major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~projected/kube-api-access-6mh76:{mountpoint:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~projected/kube-api-access-6mh76 major:0 minor:282 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config major:0 minor:280 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-tls:{mountpoint:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-tls major:0 minor:278 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~projected/kube-api-access-wdt66:{mountpoint:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~projected/kube-api-access-wdt66 major:0 minor:284 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config:{mountpoint:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config major:0 minor:277 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-tls:{mountpoint:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-tls major:0 minor:281 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~projected/kube-api-access-hjbdf:{mountpoint:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~projected/kube-api-access-hjbdf major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/e88e13a1-2eda-4ec4-968f-0bada5cf17d9/volumes/kubernetes.io~projected/kube-api-access-9n6jm:{mountpoint:/var/lib/kubelet/pods/e88e13a1-2eda-4ec4-968f-0bada5cf17d9/volumes/kubernetes.io~projected/kube-api-access-9n6jm major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd/volumes/kubernetes.io~projected/kube-api-access-tkvgs:{mountpoint:/var/lib/kubelet/pods/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd/volumes/kubernetes.io~projected/kube-api-access-tkvgs major:0 minor:399 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/5d326256623506e2717683146d6898d32d15459c126194343229f09fe6c50585/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/ebf7863eb43af4043a1a5a5d72deef5630e0de97d50c829e110cfe3e9caed705/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/3cc319a66147a36febd08a3e80f2c30ca54dacc909b6321bdf1676c05a8df55e/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/23859bc7180c50cd79c8b162f7a7d35383eb87cf85dd2b482eb6678d961d35dd/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/399979560c88f48dc61be84c8312ac501717441d53679ed114c2feb10cfb53b8/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/27a5da0b11ec4e0fc401471982ca85f3cb1c30177281d0c3fc158aafe8bb1401/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/38a59039d0759528f19af5e282364696cb8391161b145af8f991cd5aafecfed8/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/304f0d78cc44ec48ff29b6ad8e2ac3223cbf81a5868acfa8db0e1550efbe4cf7/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/a11b4e7e35100bd413fd97a4fea210f4edcfc8bed725b4d2c8e56e7778964711/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/18d5db5300606a8080f0b689f5f6e247bde11a950e9af1f8e5dc4564d67e8bf0/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-138:{mountpoint:/var/lib/containers/storage/overlay/1590b89c72f2c9d3360ca02412d388e575d798b822384f8f1370b461a8d3171f/merged major:0 minor:138 fsType:overlay blockSize:0} overlay_0-139:{mountpoint:/var/lib/containers/storage/overlay/95c821b96967c69c9a20446293b9b5243c07e2ab4b3e0a41aa4e7b83c4f114ba/merged major:0 minor:139 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/8fcb4b7d5601eb9ebd83091896be63fd7998605e9c39b1683fa3c5a71a6f32ac/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-154:{mountpoint:/var/lib/containers/storage/overlay/4979a1d42127646376a1758e8682ea2612a9c5ab7471d63187c2bd4a7f63a287/merged major:0 minor:154 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/2ea64427dff03dd77b119d46cfa6baa2fbc98eceb002ffd0d142f71bab083086/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/a7b6b6426a57cb03dafda434555c1a0fed77ae0cb0b7f5f92e43a6f89e750d1e/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/63300cf5ec06ebeb2fb35e4b1f5cea708657015ea3db3dbc5ca7dddbe61f3901/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/f2cd2a42319acf2d7293c7927ddcaed06df6f35a7f5957fb1f3b19bdf31ad64f/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/c6d595efe1694aacb2f3218f361a89f9dd1f9bddc8ccdabc80c63b6f7d40f475/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/e46506c382b3bb73e6a0b290d127d11d2bebd1edf368b98bfc0d2a667b77f31b/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/c29368dcbf6dfb51ee6a7b8788e2be330ec9a965bf7af63ce709c238a54bdd95/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-211:{mountpoint:/var/lib/containers/storage/overlay/9cb5145b5ebd306ca7c20df3e3c28595ced84f11031c3743cb75df33693a37fe/merged major:0 minor:211 fsType:overlay blockSize:0} overlay_0-213:{mountpoint:/var/lib/containers/storage/overlay/6fe44cd0e591c359ac85760b54f2dc245e2a294dcfe2de7c9213c3025ea25226/merged major:0 minor:213 fsType:overlay blockSize:0} overlay_0-215:{mountpoint:/var/lib/containers/storage/overlay/f46a6d2659cde8845c1aa8cdeec617cc32d4ccd65bf1335010359626d0b88afe/merged major:0 minor:215 fsType:overlay blockSize:0} overlay_0-217:{mountpoint:/var/lib/containers/storage/overlay/ce647955eab5eab13e593334bbc63577b09dd8c66b26938d41c7ee5d939ac56e/merged major:0 minor:217 fsType:overlay blockSize:0} overlay_0-219:{mountpoint:/var/lib/containers/storage/overlay/bb078d26c7724920d6ba0b5b2b1265c67c2224d972b48b8ec1543b179c4d2beb/merged major:0 minor:219 fsType:overlay blockSize:0} overlay_0-225:{mountpoint:/var/lib/containers/storage/overlay/4e6a64fc934797e4bcb24ef373f1efcaae6e200bd0327dc9fcf0a6a89374d584/merged major:0 minor:225 fsType:overlay blockSize:0} overlay_0-230:{mountpoint:/var/lib/containers/storage/overlay/8260c76d45cc76f910cb9bab7f79db260f8e865277e938e3943b560f9c5f87bb/merged major:0 minor:230 fsType:overlay blockSize:0} overlay_0-236:{mountpoint:/var/lib/containers/storage/overlay/80e88bbb30865738842fb1cb1c8adf0aacbf71168571955c7d050b47f5574ea8/merged major:0 minor:236 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/cc228c94c4d4f629d1288c6f6b4fca4d7e1db0d0d98b15d435ba48437094b26f/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-251:{mountpoint:/var/lib/containers/storage/overlay/56000eece3c959a5df1d4c2e6caeab27124e48d3beaabd3ece077d489baf42b2/merged major:0 minor:251 fsType:overlay blockSize:0} overlay_0-262:{mountpoint:/var/lib/containers/storage/overlay/0e9c61d18650d55ef2f711cb8493be9fe5a552dea5d8102708bb778209095b22/merged major:0 minor:262 fsType:overlay blockSize:0} overlay_0-264:{mountpoint:/var/lib/containers/storage/overlay/ac0cb62cb6da48a024cc9a9e2e31e67a81f8295d0c87efec8060e0b2d94c1a5a/merged major:0 minor:264 fsType:overlay blockSize:0} overlay_0-266:{mountpoint:/var/lib/containers/storage/overlay/8123d178fd211b29fc5bb2d21b935723984f9eabd2093d91a97a5d416124d7fc/merged major:0 minor:266 fsType:overlay blockSize:0} overlay_0-291:{mountpoint:/var/lib/containers/storage/overlay/72a5568aac4f0e35ee91c04f73894a6f1cddd60ece79e5b60779875bc10af38b/merged major:0 minor:291 fsType:overlay blockSize:0} overlay_0-293:{mountpoint:/var/lib/containers/storage/overlay/43d8952fad93c728c5aa265345fd4dba689fb1200bc53701529fa4d8ceabb7c0/merged major:0 minor:293 fsType:overlay blockSize:0} overlay_0-295:{mountpoint:/var/lib/containers/storage/overlay/1556594d3dfc577d0cd19c19bf0a7057e833b7cd68d0dc1b336c884043228a91/merged major:0 minor:295 fsType:overlay blockSize:0} overlay_0-297:{mountpoint:/var/lib/containers/storage/overlay/9b4e73dde0f71abae6be673f187a365c5eb2a822d3aaf9959bd80a4184854b01/merged major:0 minor:297 fsType:overlay blockSize:0} overlay_0-302:{mountpoint:/var/lib/containers/storage/overlay/1639b651920310d44549fd83726eb8e9947440941d2c8288ff1d0f7a1ecf51ff/merged major:0 minor:302 fsType:overlay blockSize:0} overlay_0-318:{mountpoint:/var/lib/containers/storage/overlay/aef7dd230d6c647de97014cc3a50575d789072c12741ac3e678274e7bcf1e6b3/merged major:0 minor:318 fsType:overlay blockSize:0} overlay_0-320:{mountpoint:/var/lib/containers/storage/overlay/c08873b07878a0fa3b6b0829f7d023bcd1ca23727c526f28e34c6c3a7aab7f42/merged major:0 minor:320 fsType:overlay blockSize:0} overlay_0-322:{mountpoint:/var/lib/containers/storage/overlay/bbfa11edfab32997060119f4856254ee998ebdef9101d2fbe6360289d7a8e6de/merged major:0 minor:322 fsType:overlay blockSize:0} overlay_0-329:{mountpoint:/var/lib/containers/storage/overlay/d1cf7200bdcc20a0483a056f2fc6d6cbe842c7273769ea48429f1974ae8d8be0/merged major:0 minor:329 fsType:overlay blockSize:0} overlay_0-334:{mountpoint:/var/lib/containers/storage/overlay/a456c4bbb4db70723625b47794f2a795b5f9d9bf32cf8b3c071d5b16e601291c/merged major:0 minor:334 fsType:overlay blockSize:0} overlay_0-339:{mountpoint:/var/lib/containers/storage/overlay/2b84a110bc4ebcf1fe4e37f628159fd4b597466b69cce478439e8c0bdf1203dd/merged major:0 minor:339 fsType:overlay blockSize:0} overlay_0-345:{mountpoint:/var/lib/containers/storage/overlay/90b39e1e840605dfadac862d43e246929112ef0ad12ca1d4577606e374c41443/merged major:0 minor:345 fsType:overlay blockSize:0} overlay_0-350:{mountpoint:/var/lib/containers/storage/overlay/6e297cf20ece70295c006f2d33dc0af402a45776480ad8ac44163b38348389ae/merged major:0 minor:350 fsType:overlay blockSize:0} overlay_0-355:{mountpoint:/var/lib/containers/storage/overlay/ebf16ff26916e070eac181f531824980d568139e4ef29a14bcbe3a556e426d4f/merged major:0 minor:355 fsType:overlay blockSize:0} overlay_0-362:{mountpoint:/var/lib/containers/storage/overlay/d2d3d99bc446f5591c0f3c98e9ae06d597a0fc6c82618fd93970e798e566a036/merged major:0 minor:362 fsType:overlay blockSize:0} overlay_0-366:{mountpoint:/var/lib/containers/storage/overlay/534fb2b5bb1e0f8fa169320ef3af94fd650d366e4ec0f28023548209ff541e92/merged major:0 minor:366 fsType:overlay blockSize:0} overlay_0-371:{mountpoint:/var/lib/containers/storage/overlay/8be0602126dd7fa36432261d7299eebb6cb8b14701d519cf324209bcac03bf70/merged major:0 minor:371 fsType:overlay blockSize:0} overlay_0-376:{mountpoint:/var/lib/containers/storage/overlay/2fe5f96e7e27a68cb85420d8e430347c29923d137af38c1615deb833e88bd24f/merged major:0 minor:376 fsType:overlay blockSize:0} overlay_0-389:{mountpoint:/var/lib/containers/storage/overlay/b9ee80710ffc6da7e3a6bac8cb650474b463fa4e3dd3056202a6604625164fad/merged major:0 minor:389 fsType:overlay blockSize:0} overlay_0-391:{mountpoint:/var/lib/containers/storage/overlay/bb4c32468d1542e37de137e94bdd4fbbf4c4bb38d75b7225088c2e94dcdb4e08/merged major:0 minor:391 fsType:overlay blockSize:0} overlay_0-406:{mountpoint:/var/lib/containers/storage/overlay/7426e5627959880212705ceee05df4d88b93388a6dec606a5e74016b69c0cdc8/merged major:0 minor:406 fsType:overlay blockSize:0} overlay_0-408:{mountpoint:/var/lib/containers/storage/overlay/ab3a2a30e9d5fac638547cd30d0a80dcd599249da01461eca10b1040ff3f85f2/merged major:0 minor:408 fsType:overlay blockSize:0} overlay_0-410:{mountpoint:/var/lib/containers/storage/overlay/d9ba953b99de4ea1d8ae77ef87d75c9d5bcdc11182bf3fd8697bb8f05bf3cfbd/merged major:0 minor:410 fsType:overlay blockSize:0} overlay_0-412:{mountpoint:/var/lib/containers/storage/overlay/5224d4c1e396028b100ea277c94bcdfdb076bd6673c66ce9f5ae793d938c5385/merged major:0 minor:412 fsType:overlay blockSize:0} overlay_0-414:{mountpoint:/var/lib/containers/storage/overlay/703c1f27f3b2c302662f8fef9c6bd40d6b2836ef6fd889ad7186feaa16bf526e/merged major:0 minor:414 fsType:overlay blockSize:0} overlay_0-420:{mountpoint:/var/lib/containers/storage/overlay/ec1ad5ccbe6723ac9a87918515d2ed8d317c0241856e8cfa7227fda07f2dad19/merged major:0 minor:420 fsType:overlay blockSize:0} overlay_0-425:{mountpoint:/var/lib/containers/storage/overlay/406d9096e5479643e5b7c2032086c22c32b88bd0b1ad32c5039a865881191081/merged major:0 minor:425 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/c89509962827427b92a0e696920b04363b8d6d19b4c31d35cb1192f344c311d3/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/b4653a0de61736ac2220bed1ec2702510b17a9cab3fd915cd4f4454a05167e6c/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/839292a599b8f94d7e32993181f5b4a224fced6e80125ea671270194d8d863cb/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-73:{mountpoint:/var/lib/containers/storage/overlay/58d0bbb089fa23ba05fa61e001635ed79e0358cc667e26e52b599889963675d0/merged major:0 minor:73 fsType:overlay blockSize:0} overlay_0-77:{mountpoint:/var/lib/containers/storage/overlay/39193628a4ca4caa869c27e3b780e0c17bcb0d9ad5cb7bffa04a75a774128750/merged major:0 minor:77 fsType:overlay blockSize:0} overlay_0-80:{mountpoint:/var/lib/containers/storage/overlay/b91fe442123267aa57ea2dbd789c64218fad0f807885248b8977e5039af1abc1/merged major:0 minor:80 fsType:overlay blockSize:0} overlay_0-84:{mountpoint:/var/lib/containers/storage/overlay/ece90c3c86be09ea4115cef78bed274bcfc06113f6746d42fe9a2b5f32cb3759/merged major:0 minor:84 fsType:overlay blockSize:0} overlay_0-88:{mountpoint:/var/lib/containers/storage/overlay/7ce15ac1238de3d5ba2f9413cac89b5e02928af60cdb4ddd57fac2275609b54a/merged major:0 minor:88 fsType:overlay blockSize:0} overlay_0-92:{mountpoint:/var/lib/containers/storage/overlay/1a440a6a7c7b71f60ddae4b9bfc75290e731a490e83bc0566861ff98de7ffc4a/merged major:0 minor:92 fsType:overlay blockSize:0} overlay_0-96:{mountpoint:/var/lib/containers/storage/overlay/9feb40ab26f088db73663d886c17f51f5ec504d7f2ae51d06f839395444bc0a1/merged major:0 minor:96 fsType:overlay blockSize:0}] Oct 30 23:54:38.027446 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.026996 6234 manager.go:217] Machine: {Timestamp:2026-10-30 23:54:38.026485723 +0000 UTC m=+0.059581783 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100038 MemoryCapacity:32812007424 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec28a041fd69eadefa861aba8e3828ad SystemUUID:ec28a041-fd69-eade-fa86-1aba8e3828ad BootID:d270d178-079e-470b-8da8-9a651a894d7f Filesystems:[{Device:overlay_0-73 DeviceMajor:0 DeviceMinor:73 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~projected/kube-api-access-g25jl DeviceMajor:0 DeviceMinor:203 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/169dc024bb3e3ccceaa871f8986e5f1d0354d41435f58fc2f63db5238a70ce64/userdata/shm DeviceMajor:0 DeviceMinor:209 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~projected/kube-api-access-6mh76 DeviceMajor:0 DeviceMinor:282 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-219 DeviceMajor:0 DeviceMinor:219 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:245 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-262 DeviceMajor:0 DeviceMinor:262 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:overlay_0-213 DeviceMajor:0 DeviceMinor:213 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8/userdata/shm DeviceMajor:0 DeviceMinor:260 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/cluster-tls-config DeviceMajor:0 DeviceMinor:305 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c288133571cd2a1b2dd6284bd983686e1680f74bedba7551ac110505959d2baa/userdata/shm DeviceMajor:0 DeviceMinor:97 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/7b6ce1f84777eba8918cb1c0586dd5045d9bb833cf8f678ae4bce1b424b513a9/userdata/shm DeviceMajor:0 DeviceMinor:287 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-291 DeviceMajor:0 DeviceMinor:291 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-297 DeviceMajor:0 DeviceMinor:297 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/kube-api-access-lsh47 DeviceMajor:0 DeviceMinor:317 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-329 DeviceMajor:0 DeviceMinor:329 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-406 DeviceMajor:0 DeviceMinor:406 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-138 DeviceMajor:0 DeviceMinor:138 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/config-volume DeviceMajor:0 DeviceMinor:312 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~projected/kube-api-access-hgw25 DeviceMajor:0 DeviceMinor:386 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c97249d546434a48bb4612f4f2e8ba46b234fd76e9e7f7a22671035cc42a440d/userdata/shm DeviceMajor:0 DeviceMinor:75 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-211 DeviceMajor:0 DeviceMinor:211 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-tls DeviceMajor:0 DeviceMinor:281 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-334 DeviceMajor:0 DeviceMinor:334 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-389 DeviceMajor:0 DeviceMinor:389 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406003712 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4fb7068d4de2344c1b5a0a45ca85ff582f1cef71292fbf28a343946084ac2665/userdata/shm DeviceMajor:0 DeviceMinor:94 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fa1585bc76ac004a13f526c7a9908c0b6093f7d73f687ad74b63be6d493dd629/userdata/shm DeviceMajor:0 DeviceMinor:207 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~projected/kube-api-access-f6f7l DeviceMajor:0 DeviceMinor:246 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~projected/kube-api-access-pls9z DeviceMajor:0 DeviceMinor:283 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-318 DeviceMajor:0 DeviceMinor:318 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~projected/kube-api-access-hjbdf DeviceMajor:0 DeviceMinor:64 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-139 DeviceMajor:0 DeviceMinor:139 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-251 DeviceMajor:0 DeviceMinor:251 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-metric DeviceMajor:0 DeviceMinor:309 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/alertmanager/9 DeviceMajor:0 DeviceMinor:314 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/eaff7fe2c191fda5c58eaa84cf20cc1d33dc3c59b0ee61d60c84329dbdf5ba0c/userdata/shm DeviceMajor:0 DeviceMinor:90 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-main-tls DeviceMajor:0 DeviceMinor:313 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6167cea2ee6720d185cc7d9d63357ae784f8454bbc1514d50da6af09478726d4/userdata/shm DeviceMajor:0 DeviceMinor:400 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/52a55c557b0b47b9214e5e2d657188bb038f703aa463ed5bfb8c94bda88d30f3/userdata/shm DeviceMajor:0 DeviceMinor:402 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-215 DeviceMajor:0 DeviceMinor:215 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-tls DeviceMajor:0 DeviceMinor:278 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-366 DeviceMajor:0 DeviceMinor:366 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:56 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~projected/kube-api-access-cb42l DeviceMajor:0 DeviceMinor:65 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2d8e482a7489e7c1d8bb25e103ce9c5a4d63006df4754b09107af3a7e6adb454/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-293 DeviceMajor:0 DeviceMinor:293 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/web-config/init-config-reloader/6 DeviceMajor:0 DeviceMinor:314 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:57 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-154 DeviceMajor:0 DeviceMinor:154 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~secret/insights-runtime-extractor-tls DeviceMajor:0 DeviceMinor:197 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/web-config DeviceMajor:0 DeviceMinor:314 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-412 DeviceMajor:0 DeviceMinor:412 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/e88e13a1-2eda-4ec4-968f-0bada5cf17d9/volumes/kubernetes.io~projected/kube-api-access-9n6jm DeviceMajor:0 DeviceMinor:63 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-295 DeviceMajor:0 DeviceMinor:295 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-376 DeviceMajor:0 DeviceMinor:376 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-420 DeviceMajor:0 DeviceMinor:420 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-266 DeviceMajor:0 DeviceMinor:266 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-350 DeviceMajor:0 DeviceMinor:350 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-371 DeviceMajor:0 DeviceMinor:371 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-425 DeviceMajor:0 DeviceMinor:425 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-tls DeviceMajor:0 DeviceMinor:279 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f1d9c92f87b221ae6448aa10ffb6a587a2be53b5e62833d4354e231cea180ec9/userdata/shm DeviceMajor:0 DeviceMinor:324 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:385 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4d1f10bd-b2d0-486d-9caa-21b650348f93/volumes/kubernetes.io~secret/metrics-certs DeviceMajor:0 DeviceMinor:398 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-80 DeviceMajor:0 DeviceMinor:80 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~empty-dir/config-out DeviceMajor:0 DeviceMinor:311 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~projected/tls-assets DeviceMajor:0 DeviceMinor:315 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406003712 Type:vfs Inodes:1048576 HasInodes:true} {Device:overlay_0-236 DeviceMajor:0 DeviceMinor:236 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/4434ed6e397b33bdf4c842565a7a451c8edb82b191db917fb3674759d2732981/userdata/shm DeviceMajor:0 DeviceMinor:404 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-225 DeviceMajor:0 DeviceMinor:225 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/2fb44b25eac733c2215a7258808110ebd732d31b0682576ed31ed741c048c9f5/userdata/shm DeviceMajor:0 DeviceMinor:285 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-302 DeviceMajor:0 DeviceMinor:302 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:381 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-339 DeviceMajor:0 DeviceMinor:339 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-96 DeviceMajor:0 DeviceMinor:96 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f/userdata/shm DeviceMajor:0 DeviceMinor:387 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-77 DeviceMajor:0 DeviceMinor:77 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-84 DeviceMajor:0 DeviceMinor:84 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-217 DeviceMajor:0 DeviceMinor:217 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11/userdata/shm DeviceMajor:0 DeviceMinor:249 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-322 DeviceMajor:0 DeviceMinor:322 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/827a07c8-e408-4526-a58b-4f7982a7dcaa/volumes/kubernetes.io~projected/kube-api-access-8z54h DeviceMajor:0 DeviceMinor:59 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/volumes/kubernetes.io~projected/kube-api-access-9b787 DeviceMajor:0 DeviceMinor:60 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/44459cea-2fa8-44fb-ad9f-9aeafcdae6df/volumes/kubernetes.io~projected/kube-api-access-9nd8n DeviceMajor:0 DeviceMinor:66 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-264 DeviceMajor:0 DeviceMinor:264 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~secret/node-exporter-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:277 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-345 DeviceMajor:0 DeviceMinor:345 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-410 DeviceMajor:0 DeviceMinor:410 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:58 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-391 DeviceMajor:0 DeviceMinor:391 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-408 DeviceMajor:0 DeviceMinor:408 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b9215d31-9f16-4011-a873-2995f3e65cf8/volumes/kubernetes.io~secret/openshift-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:280 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/d775e8d4-9d4a-4779-a5c5-4442e8d3a306/volumes/kubernetes.io~projected/kube-api-access-wdt66 DeviceMajor:0 DeviceMinor:284 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/14e77ba7-07ab-4933-9268-760aa9712cb8/volumes/kubernetes.io~secret/metrics-tls DeviceMajor:0 DeviceMinor:201 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volume-subpaths/cluster-tls-config/alertmanager/10 DeviceMajor:0 DeviceMinor:305 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-355 DeviceMajor:0 DeviceMinor:355 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~projected/kube-api-access-wqd5r DeviceMajor:0 DeviceMinor:259 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/fe02d1abb5f66391a3570ab134c40fd6f2c3d456ff3e63c57e95214ed1ecfc01/userdata/shm DeviceMajor:0 DeviceMinor:289 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy-web DeviceMajor:0 DeviceMinor:316 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-414 DeviceMajor:0 DeviceMinor:414 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/b71f4919-41b2-469c-af26-b016813419ec/volumes/kubernetes.io~projected/kube-api-access-58qmb DeviceMajor:0 DeviceMinor:61 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d8aa19bcacc79b064a616b9a2b71b2b068a9efbbeaa1e56fe6135dff49c93f10/userdata/shm DeviceMajor:0 DeviceMinor:86 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-92 DeviceMajor:0 DeviceMinor:92 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/47e29498c11af59e6157b868b93be33dccc10e57dbb719076dbc2fa75eac58b9/userdata/shm DeviceMajor:0 DeviceMinor:205 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:257 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/4285655f-2bda-4d2b-87f5-b9506c3fdb51/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:393 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd/volumes/kubernetes.io~projected/kube-api-access-tkvgs DeviceMajor:0 DeviceMinor:399 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/6e52b6974c1589a146c9394bcd0891386776022d94ea294a375febe02045ae1f/userdata/shm DeviceMajor:0 DeviceMinor:82 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-230 DeviceMajor:0 DeviceMinor:230 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes/kubernetes.io~secret/console-oauth-config DeviceMajor:0 DeviceMinor:258 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes/kubernetes.io~secret/secret-alertmanager-kube-rbac-proxy DeviceMajor:0 DeviceMinor:310 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/721668b5-4594-43bb-82ae-f8fd5e80fd2e/volumes/kubernetes.io~projected/kube-api-access-rzcpr DeviceMajor:0 DeviceMinor:67 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-88 DeviceMajor:0 DeviceMinor:88 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/32b07947-679b-40bb-849d-10815894262d/volumes/kubernetes.io~projected/kube-api-access-jd7df DeviceMajor:0 DeviceMinor:202 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/58a99b61-5c4d-41c8-9b7a-172aa3edc72b/volumes/kubernetes.io~projected/kube-api-access-h8cgn DeviceMajor:0 DeviceMinor:204 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/46826886-bbcb-4f1e-ae2a-94f7feb26d76/volumes/kubernetes.io~secret/kube-state-metrics-kube-rbac-proxy-config DeviceMajor:0 DeviceMinor:272 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:overlay_0-320 DeviceMajor:0 DeviceMinor:320 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-362 DeviceMajor:0 DeviceMinor:362 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8411675f-2736-483b-8a74-a15c12287ea1/volumes/kubernetes.io~projected/kube-api-access-7ctp6 DeviceMajor:0 DeviceMinor:62 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes/kubernetes.io~secret/console-serving-cert DeviceMajor:0 DeviceMinor:241 Capacity:31633408000 Type:vfs Inodes:4005372 HasInodes:true} {Device:/run/containers/storage/overlay-containers/d4fa30ddb0d36612813fd72b38d0965f7788e2c128bc9c18d26d81f8514f5e68/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005372 HasInodes:true} {Device:/var/lib/kubelet/pods/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562402304 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:169dc024bb3e3cc MacAddress:8a:4c:a0:77:2a:7e Speed:10000 Mtu:8901} {Name:2fb44b25eac733c MacAddress:fa:05:f0:a5:52:6c Speed:10000 Mtu:8901} {Name:4434ed6e397b33b MacAddress:f6:9e:96:72:96:63 Speed:10000 Mtu:8901} {Name:47e29498c11af59 MacAddress:6e:bb:2c:75:b3:da Speed:10000 Mtu:8901} {Name:52a55c557b0b47b MacAddress:46:a1:79:1a:8c:14 Speed:10000 Mtu:8901} {Name:52e0fa6449d30a4 MacAddress:42:e0:d1:a2:5b:8d Speed:10000 Mtu:8901} {Name:6167cea2ee6720d MacAddress:ba:bb:10:16:73:d3 Speed:10000 Mtu:8901} {Name:7b6ce1f84777eba MacAddress:a6:36:18:32:46:5f Speed:10000 Mtu:8901} {Name:a7018363aa7f0cc MacAddress:d6:86:06:27:94:b6 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:e1:71:e7:eb Speed:0 Mtu:9001} {Name:br-int MacAddress:ba:89:39:19:7e:0a Speed:0 Mtu:8901} {Name:e97f7b1e31c71fe MacAddress:d6:41:8c:05:f8:66 Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:e1:71:e7:eb Speed:0 Mtu:9001} {Name:f1d9c92f87b221a MacAddress:aa:26:6f:14:7f:8d Speed:10000 Mtu:8901} {Name:fa1585bc76ac004 MacAddress:e6:a1:fb:32:4a:46 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:42:2e:65:1c:da:82 Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:2a:b3:3f:c2:24:36 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812007424 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None} Oct 30 23:54:38.027660 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027452 6234 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available. Oct 30 23:54:38.027660 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027501 6234 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:} Oct 30 23:54:38.027819 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027773 6234 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[] Oct 30 23:54:38.027953 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027821 6234 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-130-152.us-east-2.compute.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2} Oct 30 23:54:38.028000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027956 6234 topology_manager.go:143] "Creating topology manager with none policy" Oct 30 23:54:38.028000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027964 6234 container_manager_linux.go:311] "Creating device plugin manager" Oct 30 23:54:38.028000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027976 6234 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:54:38.028000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.027989 6234 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock" Oct 30 23:54:38.028105 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028085 6234 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager" Oct 30 23:54:38.028259 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028249 6234 state_mem.go:41] "Initialized" logger="CPUManager state memory" Oct 30 23:54:38.028375 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028366 6234 server.go:1235] "Using root directory" path="/var/lib/kubelet" Oct 30 23:54:38.028412 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028404 6234 kubelet.go:493] "Attempting to sync node with API server" Oct 30 23:54:38.028442 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028419 6234 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests" Oct 30 23:54:38.028442 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028429 6234 file.go:69] "Watching path" path="/etc/kubernetes/manifests" Oct 30 23:54:38.028442 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028438 6234 kubelet.go:405] "Adding apiserver pod source" Oct 30 23:54:38.028540 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.028446 6234 apiserver.go:42] "Waiting for node sync before watching apiserver pods" Oct 30 23:54:38.029174 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.029161 6234 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:54:38.029174 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.029174 6234 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:54:38.029735 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.029715 6234 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1" Oct 30 23:54:38.033028 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033014 6234 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint" Oct 30 23:54:38.033028 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033028 6234 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking" Oct 30 23:54:38.033146 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033137 6234 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem" Oct 30 23:54:38.033335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033324 6234 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled" Oct 30 23:54:38.033364 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033353 6234 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled" Oct 30 23:54:38.033456 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033445 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume" Oct 30 23:54:38.033489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033466 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir" Oct 30 23:54:38.033489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033480 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033489 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033503 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033509 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033515 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033520 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033527 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033533 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap" Oct 30 23:54:38.033543 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033545 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected" Oct 30 23:54:38.033745 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033554 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume" Oct 30 23:54:38.033745 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033578 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi" Oct 30 23:54:38.033745 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.033586 6234 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image" Oct 30 23:54:38.037002 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.036989 6234 server.go:1263] "Started kubelet" Oct 30 23:54:38.037147 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.037071 6234 server.go:182] "Starting to listen" address="0.0.0.0" port=10250 Oct 30 23:54:38.037147 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.037087 6234 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10 Oct 30 23:54:38.037147 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.037180 6234 server_v1.go:49] "podresources" method="list" useActivePods=true Oct 30 23:54:38.037147 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.037584 6234 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock" Oct 30 23:54:38.037691 ip-10-0-130-152 systemd[1]: Started Kubernetes Kubelet. Oct 30 23:54:38.038535 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.038413 6234 server.go:317] "Adding debug handlers to kubelet server" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042222 6234 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042241 6234 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042298 6234 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-10-29 23:48:28 +0000 UTC" deadline="2028-04-24 10:20:50.983369493 +0000 UTC" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042318 6234 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12994h26m12.941054831s" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042344 6234 volume_manager.go:309] "The desired_state_of_world populator starts" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.042349 6234 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-130-152.us-east-2.compute.internal\" not found" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042355 6234 volume_manager.go:311] "Starting Kubelet Volume Manager" Oct 30 23:54:38.042404 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.042366 6234 desired_state_of_world_populator.go:146] "Desired state populator starts to run" Oct 30 23:54:38.043738 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.043662 6234 factory.go:55] Registering systemd factory Oct 30 23:54:38.043738 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.043687 6234 factory.go:223] Registration of the systemd container factory successfully Oct 30 23:54:38.045129 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045084 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e77ba7-07ab-4933-9268-760aa9712cb8" volumeName="kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl" seLinuxMountContext="" Oct 30 23:54:38.045129 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045122 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045137 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045152 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045166 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045178 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b71f4919-41b2-469c-af26-b016813419ec" volumeName="kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045190 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9215d31-9f16-4011-a873-2995f3e65cf8" volumeName="kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045203 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045219 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bdcd7f1-7d32-40eb-8daf-8c83e77e8296" volumeName="kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045233 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="721668b5-4594-43bb-82ae-f8fd5e80fd2e" volumeName="kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045246 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="827a07c8-e408-4526-a58b-4f7982a7dcaa" volumeName="kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045258 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e88e13a1-2eda-4ec4-968f-0bada5cf17d9" volumeName="kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm" seLinuxMountContext="" Oct 30 23:54:38.045275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045270 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045284 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045300 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045312 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae" volumeName="kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045325 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045338 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045350 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d" volumeName="kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045363 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045371 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045378 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411675f-2736-483b-8a74-a15c12287ea1" volumeName="kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045386 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045393 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045401 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a99b61-5c4d-41c8-9b7a-172aa3edc72b" volumeName="kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045408 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="827a07c8-e408-4526-a58b-4f7982a7dcaa" volumeName="kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045421 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411675f-2736-483b-8a74-a15c12287ea1" volumeName="kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045434 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae" volumeName="kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045446 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045454 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4285655f-2bda-4d2b-87f5-b9506c3fdb51" volumeName="kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045462 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44459cea-2fa8-44fb-ad9f-9aeafcdae6df" volumeName="kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045469 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d1f10bd-b2d0-486d-9caa-21b650348f93" volumeName="kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045477 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a99b61-5c4d-41c8-9b7a-172aa3edc72b" volumeName="kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045484 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a99b61-5c4d-41c8-9b7a-172aa3edc72b" volumeName="kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045492 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045499 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045506 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e77ba7-07ab-4933-9268-760aa9712cb8" volumeName="kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045514 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae" volumeName="kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045521 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="fa89d9ae-29cc-4664-ae9d-ca28b785c7bd" volumeName="kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045533 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045541 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045549 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d" volumeName="kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045556 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411675f-2736-483b-8a74-a15c12287ea1" volumeName="kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045564 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045571 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045579 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045587 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e77ba7-07ab-4933-9268-760aa9712cb8" volumeName="kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045594 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045601 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045608 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045614 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045622 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b71f4919-41b2-469c-af26-b016813419ec" volumeName="kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045629 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9215d31-9f16-4011-a873-2995f3e65cf8" volumeName="kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045637 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9215d31-9f16-4011-a873-2995f3e65cf8" volumeName="kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045644 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045651 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045664 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="721668b5-4594-43bb-82ae-f8fd5e80fd2e" volumeName="kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045671 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045679 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045686 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="58a99b61-5c4d-41c8-9b7a-172aa3edc72b" volumeName="kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045694 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44459cea-2fa8-44fb-ad9f-9aeafcdae6df" volumeName="kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045702 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="4d1f10bd-b2d0-486d-9caa-21b650348f93" volumeName="kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045709 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="5bdcd7f1-7d32-40eb-8daf-8c83e77e8296" volumeName="kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045717 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045724 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b9215d31-9f16-4011-a873-2995f3e65cf8" volumeName="kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045732 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045739 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae" volumeName="kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045751 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32b07947-679b-40bb-849d-10815894262d" volumeName="kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045759 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045766 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045777 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045821 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045832 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="14e77ba7-07ab-4933-9268-760aa9712cb8" volumeName="kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045843 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045853 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="44459cea-2fa8-44fb-ad9f-9aeafcdae6df" volumeName="kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045863 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045882 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045894 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045905 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae" volumeName="kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045915 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" volumeName="kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l" seLinuxMountContext="" Oct 30 23:54:38.045891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045925 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8411675f-2736-483b-8a74-a15c12287ea1" volumeName="kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045937 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045949 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045966 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="3027d685-5695-4d70-8759-5b28d323af4f" volumeName="kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045978 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="32b07947-679b-40bb-849d-10815894262d" volumeName="kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.045989 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="398381a0-d000-45eb-8c18-a1f6b2f2ed60" volumeName="kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046000 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="46826886-bbcb-4f1e-ae2a-94f7feb26d76" volumeName="kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046010 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" volumeName="kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046022 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="b71f4919-41b2-469c-af26-b016813419ec" volumeName="kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046032 6234 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" volumeName="kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config" seLinuxMountContext="" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046043 6234 reconstruct.go:97] "Volume reconstruction finished" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.046050 6234 reconciler.go:29] "Reconciler: start to sync state" Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047462 6234 factory.go:153] Registering CRI-O factory Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047480 6234 factory.go:223] Registration of the crio container factory successfully Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047548 6234 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047568 6234 factory.go:103] Registering Raw factory Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047583 6234 manager.go:1199] Started watching for new ooms in manager Oct 30 23:54:38.048525 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.047918 6234 manager.go:319] Starting recovery of all containers Oct 30 23:54:38.055000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.054980 6234 reconstruct.go:205] "DevicePaths of reconstructed volumes updated" Oct 30 23:54:38.057217 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.057193 6234 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:38.061114 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.061091 6234 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:38.062564 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.062530 6234 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:38.071178 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.071132 6234 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4" Oct 30 23:54:38.073481 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.073461 6234 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6" Oct 30 23:54:38.073569 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.073489 6234 status_manager.go:249] "Starting to sync pod status with apiserver" Oct 30 23:54:38.073569 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.073507 6234 kubelet.go:2521] "Starting kubelet main sync loop" Oct 30 23:54:38.073707 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.073547 6234 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]" Oct 30 23:54:38.078385 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.078355 6234 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161" Oct 30 23:54:38.100637 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.100601 6234 generic.go:356] "Generic (PLEG): container finished" podID="d775e8d4-9d4a-4779-a5c5-4442e8d3a306" containerID="8d2515a3573576b10c118226216d57c39a9f278f2ea5654313a5787804acc575" exitCode=0 Oct 30 23:54:38.104099 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.104073 6234 generic.go:356] "Generic (PLEG): container finished" podID="9d885cc10bee944101eff496e23ac512" containerID="dca40c86edaf5f621b4245ea00d4d7fe5f36c55cb96bea083d204c671602b52e" exitCode=0 Oct 30 23:54:38.109651 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.109585 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" exitCode=0 Oct 30 23:54:38.132616 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132586 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="8b6206bc0f477477ad1ad576ca2bd0e442a33cf0991645dc055739e4c9b80303" exitCode=0 Oct 30 23:54:38.132616 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132616 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="5d2d8465e4d0a0902411fc54462051c75569b569952cfeb00494cc93d47e7c0d" exitCode=0 Oct 30 23:54:38.132798 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132626 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="54d98417d2585dce8f2c3f4b030228e724968750597f9c4c5937065765f2d391" exitCode=0 Oct 30 23:54:38.132798 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132634 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="77edef1f8157f3a306bddba181df12a20b81e9adea7d47a53fadc8e05da695d2" exitCode=0 Oct 30 23:54:38.132906 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132645 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="7b4628bdaff3a1a5c2c0ce818cb518a1219c888f9a4e45dd4403761d8c77082a" exitCode=0 Oct 30 23:54:38.132906 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.132864 6234 generic.go:356] "Generic (PLEG): container finished" podID="8411675f-2736-483b-8a74-a15c12287ea1" containerID="964eab2561fdd6c4b1534d83fefb94bd281e960a374a92a30163edc6eb76a677" exitCode=0 Oct 30 23:54:38.150737 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.150711 6234 manager.go:324] Recovery completed Oct 30 23:54:38.174247 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174154 6234 cpu_manager.go:226] "Starting" policy="none" Oct 30 23:54:38.174247 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174187 6234 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s" Oct 30 23:54:38.174247 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174206 6234 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory" Oct 30 23:54:38.174247 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.174205 6234 kubelet.go:2545] "Skipping pod synchronization" err="container runtime status check may not have completed yet" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174362 6234 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet="" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174378 6234 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={} Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174401 6234 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174411 6234 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet="" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174419 6234 policy_none.go:50] "Start" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174432 6234 memory_manager.go:187] "Starting memorymanager" policy="None" Oct 30 23:54:38.174518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.174446 6234 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint" Oct 30 23:54:38.176127 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.176112 6234 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint" Oct 30 23:54:38.176195 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.176130 6234 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint" Oct 30 23:54:38.176195 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.176141 6234 policy_none.go:44] "Start" Oct 30 23:54:38.180653 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.180628 6234 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"} Oct 30 23:54:38.183127 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.183114 6234 manager.go:341] "Starting Device Plugin manager" Oct 30 23:54:38.183698 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.183137 6234 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint" Oct 30 23:54:38.183738 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.183704 6234 server.go:87] "Starting device plugin registration server" Oct 30 23:54:38.183994 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.183983 6234 eviction_manager.go:194] "Eviction manager: starting control loop" Oct 30 23:54:38.184037 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.183995 6234 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s" Oct 30 23:54:38.184108 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.184090 6234 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry" Oct 30 23:54:38.184219 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.184192 6234 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock" Oct 30 23:54:38.184275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.184230 6234 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts" Oct 30 23:54:38.184275 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.184237 6234 plugin_manager.go:121] "Starting Kubelet Plugin Manager" Oct 30 23:54:38.184354 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.184294 6234 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-10-30T23:54:38.184218251Z","UUID":"2129511d-cf14-4447-9b0d-9d1d6346286d","Handler":null,"Name":"","Endpoint":""} Oct 30 23:54:38.185665 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.185642 6234 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\"" Oct 30 23:54:38.185741 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.185652 6234 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0 Oct 30 23:54:38.185741 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.185730 6234 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock Oct 30 23:54:38.284962 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.284924 6234 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach" Oct 30 23:54:38.286938 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.286920 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientMemory" Oct 30 23:54:38.287038 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.286953 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasNoDiskPressure" Oct 30 23:54:38.287038 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.286965 6234 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-130-152.us-east-2.compute.internal" event="NodeHasSufficientPID" Oct 30 23:54:38.287038 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.287010 6234 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.303021 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.302989 6234 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.303144 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.303087 6234 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.374843 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.374773 6234 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal","kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal"] Oct 30 23:54:38.375650 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375437 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"4e44186c11950043d5117909b34eae0d","Type":"ContainerStarted","Data":"c876dcc48c64642c4cd589acaea644f9a8a615e87b779b3ebab7e4f7377859a8"} Oct 30 23:54:38.375650 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375593 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.375847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375670 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"4e44186c11950043d5117909b34eae0d","Type":"ContainerStarted","Data":"ac0c1068eecf422224d188c592dff4304b358d96bcf93aac478b8df2acc3b4e5"} Oct 30 23:54:38.375847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375717 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerStarted","Data":"938af9deaef2c3605938b8cc551be2e8a315350d09687ae9fb64fecbab6089c7"} Oct 30 23:54:38.375847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375737 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerDied","Data":"dca40c86edaf5f621b4245ea00d4d7fe5f36c55cb96bea083d204c671602b52e"} Oct 30 23:54:38.375847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375756 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" event={"ID":"9d885cc10bee944101eff496e23ac512","Type":"ContainerStarted","Data":"3c057b4141191c7084a07727f7c4d179b9091c5f72039d75217e41f000effcdd"} Oct 30 23:54:38.375847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.375837 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.391714 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.391688 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:38.391857 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.391768 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.393604 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.393591 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:38.393673 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:38.393623 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.457020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.456977 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.457020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.457023 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.457216 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.457044 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557589 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557558 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557681 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557596 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557681 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557619 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557681 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557665 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/4e44186c11950043d5117909b34eae0d-config\") pod \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"4e44186c11950043d5117909b34eae0d\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557681 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557667 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:38.557848 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:38.557666 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/9d885cc10bee944101eff496e23ac512-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" (UID: \"9d885cc10bee944101eff496e23ac512\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:39.028956 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.028912 6234 apiserver.go:52] "Watching apiserver" Oct 30 23:54:39.048356 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.048327 6234 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66" Oct 30 23:54:39.049120 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.049094 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal","openshift-dns/node-resolver-q7jkq","openshift-multus/network-metrics-daemon-6dpdv","openshift-cluster-node-tuning-operator/tuned-bsxgt","openshift-console/console-6799bcd5d4-r6r54","openshift-image-registry/node-ca-4qqrz","openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7","kube-system/global-pull-secret-syncer-45ssd","openshift-console/console-b4d4f6d76-jd8lq","openshift-ingress-canary/ingress-canary-5sqx2","openshift-multus/multus-rblpn","openshift-insights/insights-runtime-extractor-vscwx","openshift-dns/dns-default-dw7gl","openshift-monitoring/alertmanager-main-0","openshift-multus/multus-additional-cni-plugins-n72w7","kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal","openshift-console/console-588d4889bc-vs299","openshift-ovn-kubernetes/ovnkube-node-lqv9d","openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n","openshift-network-diagnostics/network-check-target-8nvcl","openshift-monitoring/node-exporter-q2kgm","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp","kube-system/konnectivity-agent-dbh42","openshift-network-operator/iptables-alerter-dx424"] Oct 30 23:54:39.054674 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.054649 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\"" Oct 30 23:54:39.054674 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.054661 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-z44sw\"" Oct 30 23:54:39.055273 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.055252 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wjrgq\"" Oct 30 23:54:39.056678 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.056656 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" Oct 30 23:54:39.056847 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.056833 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-tzf9l\"" Oct 30 23:54:39.062767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.062747 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.062767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.062748 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.062952 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.062919 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.063532 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.063500 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.063614 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.063568 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" Oct 30 23:54:39.066443 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.066426 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.066913 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.066893 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.066913 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.066894 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.068280 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.068268 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.068628 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.068608 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.069073 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.069055 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.069073 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.069057 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.069666 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.069653 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.070906 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.070886 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-w9wmr\"" Oct 30 23:54:39.071030 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071014 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-9gq6b\"" Oct 30 23:54:39.071109 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071014 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-tb4c7\"" Oct 30 23:54:39.071109 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071025 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" Oct 30 23:54:39.071221 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071037 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" Oct 30 23:54:39.071365 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071351 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" Oct 30 23:54:39.071556 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071531 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.071556 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071545 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" Oct 30 23:54:39.071649 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071568 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.071740 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.071720 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.073714 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.073697 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" Oct 30 23:54:39.073714 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.073702 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" Oct 30 23:54:39.076011 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.075899 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" Oct 30 23:54:39.076011 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.075920 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" Oct 30 23:54:39.076011 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.075954 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-xhccc\"" Oct 30 23:54:39.076011 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.075962 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-4mxgc\"" Oct 30 23:54:39.076280 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076230 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" Oct 30 23:54:39.076280 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076241 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-q6gjg\"" Oct 30 23:54:39.076280 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076253 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-427wx\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076290 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-4xs9v\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076302 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-d2wwf\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076331 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076332 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-dockercfg-fptsc\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076356 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-d9xt4\"" Oct 30 23:54:39.076440 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076397 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-4k95d\"" Oct 30 23:54:39.076724 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076433 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" Oct 30 23:54:39.076724 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076490 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" Oct 30 23:54:39.076724 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076502 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.076724 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076534 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\"" Oct 30 23:54:39.076724 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.076557 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" Oct 30 23:54:39.077105 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.077087 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" Oct 30 23:54:39.079018 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.079001 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.082865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.082851 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:54:39.083011 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.082993 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-tls\"" Oct 30 23:54:39.083097 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083061 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:54:39.083184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083093 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-qtpsm\"" Oct 30 23:54:39.083184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083119 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:54:39.083184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083131 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:54:39.083184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083135 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:54:39.083184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083119 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:54:39.083639 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.083625 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-g4klp\"" Oct 30 23:54:39.084346 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084333 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" Oct 30 23:54:39.084389 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084347 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" Oct 30 23:54:39.084602 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084591 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:54:39.084654 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084633 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-n9slr\"" Oct 30 23:54:39.084706 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084660 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:54:39.084706 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084678 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\"" Oct 30 23:54:39.084706 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084687 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\"" Oct 30 23:54:39.084706 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084702 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" Oct 30 23:54:39.084706 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.084707 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-mjqb7\"" Oct 30 23:54:39.085097 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.085080 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wcgjs\"" Oct 30 23:54:39.085140 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.085098 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" Oct 30 23:54:39.085196 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.085185 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:54:39.137048 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.137021 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:39.137214 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.137162 6234 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:39.144658 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.144637 6234 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world" Oct 30 23:54:39.152921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.152902 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:39.153026 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:39.152953 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:39.160226 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160206 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.160290 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160237 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.160290 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160256 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.160290 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160272 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.160390 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160330 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nd8n\" (UniqueName: \"kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.160428 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.160523 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160504 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.160576 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160539 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/721668b5-4594-43bb-82ae-f8fd5e80fd2e-iptables-alerter-script\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.160645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160574 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.160645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160600 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.160645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160609 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.160645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160629 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.160849 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160709 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.160849 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160742 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wdt66\" (UniqueName: \"kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.160981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160847 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.160981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160892 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.160981 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.160954 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.161077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161004 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.161077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161023 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.161077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161039 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.161077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161059 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.161228 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161076 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzcpr\" (UniqueName: \"kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.161228 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161092 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.161228 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161123 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovn-node-metrics-cert\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.161228 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161128 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.161228 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161225 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161268 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161309 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161315 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161348 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:39.161448 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161407 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-env-overrides\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161501 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161551 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161555 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/4285655f-2bda-4d2b-87f5-b9506c3fdb51-original-pull-secret\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/4d1f10bd-b2d0-486d-9caa-21b650348f93-metrics-certs\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:39.161662 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161599 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.161873 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161681 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.161873 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161715 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7ctp6\" (UniqueName: \"kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.161873 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161727 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-tls\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.161873 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161773 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162008 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161913 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162008 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.162008 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161951 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.162008 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161977 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.162008 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.161983 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.162165 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162008 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.162165 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162015 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162165 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162043 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162165 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162152 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162163 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-script-lib\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162182 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-kube-rbac-proxy-config\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162205 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162266 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162284 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162301 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjbdf\" (UniqueName: \"kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162303 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162326 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162322 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162345 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162351 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162398 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9n6jm\" (UniqueName: \"kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162439 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/b9215d31-9f16-4011-a873-2995f3e65cf8-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162507 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162538 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/32b07947-679b-40bb-849d-10815894262d-cert\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162539 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162614 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162661 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.162761 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162756 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162832 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162857 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162873 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162897 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162922 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162955 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14e77ba7-07ab-4933-9268-760aa9712cb8-tmp-dir\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162965 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.162999 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163034 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163087 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163096 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-daemon-config\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163098 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163118 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.163193 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163165 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163205 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163215 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163247 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163331 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8cgn\" (UniqueName: \"kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163356 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163362 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163366 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-data-volume\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163374 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163407 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163423 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163453 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163471 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163568 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163597 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163613 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163622 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163648 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163677 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163703 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163729 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163749 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.163805 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163767 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163827 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163907 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163947 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/14e77ba7-07ab-4933-9268-760aa9712cb8-metrics-tls\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163958 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.163971 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164000 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164025 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8z54h\" (UniqueName: \"kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164035 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-etc-tuned\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164039 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-tls\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164052 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164077 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164086 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164103 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pls9z\" (UniqueName: \"kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164185 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164228 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164247 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164270 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164274 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164286 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164307 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164331 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164355 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164374 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164388 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164402 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164418 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164423 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164455 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b787\" (UniqueName: \"kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164488 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164514 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cni-binary-copy\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164545 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-agent-certs\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164591 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164647 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164691 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164728 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164741 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/5bdcd7f1-7d32-40eb-8daf-8c83e77e8296-konnectivity-ca\") pod \"konnectivity-agent-dbh42\" (UID: \"5bdcd7f1-7d32-40eb-8daf-8c83e77e8296\") " pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:54:39.164767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164774 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6mh76\" (UniqueName: \"kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164828 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-tmp-dir\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164835 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164827 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/b71f4919-41b2-469c-af26-b016813419ec-tmp\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164871 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164893 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164914 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164962 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.164997 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165034 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165057 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cb42l\" (UniqueName: \"kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165094 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165174 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165209 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165235 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165268 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g25jl\" (UniqueName: \"kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165311 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165349 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165453 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165457 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"volume-directive-shadow\" (UniqueName: \"kubernetes.io/empty-dir/46826886-bbcb-4f1e-ae2a-94f7feb26d76-volume-directive-shadow\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165467 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/8411675f-2736-483b-8a74-a15c12287ea1-cni-binary-copy\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165494 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58qmb\" (UniqueName: \"kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165514 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165530 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165550 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165627 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165731 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165770 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165844 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165870 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165887 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165899 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165925 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165949 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165973 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd7df\" (UniqueName: \"kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165986 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165994 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-textfile\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165999 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.165869 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166015 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166047 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166076 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166104 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166132 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166179 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166293 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/827a07c8-e408-4526-a58b-4f7982a7dcaa-serviceca\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166319 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166335 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.166406 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166379 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-ovnkube-config\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.167822 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.166513 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.179382 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179362 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.179664 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179647 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\"" Oct 30 23:54:39.179664 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179670 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\"" Oct 30 23:54:39.179664 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179670 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\"" Oct 30 23:54:39.179965 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179748 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.179965 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179822 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.179965 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179902 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.179965 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.179945 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" Oct 30 23:54:39.180184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.180092 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.180184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.180096 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\"" Oct 30 23:54:39.180184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.180123 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" Oct 30 23:54:39.180184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.180131 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" Oct 30 23:54:39.181488 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.181465 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.181918 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.181896 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" Oct 30 23:54:39.181918 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.181893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/14e77ba7-07ab-4933-9268-760aa9712cb8-config-volume\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.182078 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.181942 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.182136 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.182070 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.183553 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.183535 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.184505 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.184485 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.184578 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.184518 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.186067 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.186048 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.186163 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.186050 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.186163 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.186080 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.186163 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.186130 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-accelerators-collector-config\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.186269 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.186256 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.187115 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.187095 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" Oct 30 23:54:39.187195 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.187112 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-state-metrics-custom-resource-state-configmap\"" Oct 30 23:54:39.187245 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.187232 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"trusted-ca-bundle\"" Oct 30 23:54:39.190687 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.190668 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-metrics-client-ca\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.190871 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.190856 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-metrics-client-ca\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.191486 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.191471 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b9215d31-9f16-4011-a873-2995f3e65cf8-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.193737 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.193720 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:54:39.194866 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.194849 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.194974 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.194956 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.195206 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.195190 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.195259 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.195215 6234 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]" Oct 30 23:54:39.195308 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:39.195268 6234 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-130-152.us-east-2.compute.internal" Oct 30 23:54:39.196386 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.196371 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-state-metrics-custom-resource-state-configmap\" (UniqueName: \"kubernetes.io/configmap/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-state-metrics-custom-resource-state-configmap\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.202744 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.202723 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.246481 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.246444 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9n6jm\" (UniqueName: \"kubernetes.io/projected/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kube-api-access-9n6jm\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.246628 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.246610 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"console-b4d4f6d76-jd8lq\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:54:39.246904 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.246886 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6mh76\" (UniqueName: \"kubernetes.io/projected/b9215d31-9f16-4011-a873-2995f3e65cf8-kube-api-access-6mh76\") pod \"openshift-state-metrics-7bd7489957-jfc9n\" (UID: \"b9215d31-9f16-4011-a873-2995f3e65cf8\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-jfc9n" Oct 30 23:54:39.247028 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.247010 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g25jl\" (UniqueName: \"kubernetes.io/projected/14e77ba7-07ab-4933-9268-760aa9712cb8-kube-api-access-g25jl\") pod \"dns-default-dw7gl\" (UID: \"14e77ba7-07ab-4933-9268-760aa9712cb8\") " pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:39.249115 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.249099 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"console-6799bcd5d4-r6r54\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:54:39.256161 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.253118 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzcpr\" (UniqueName: \"kubernetes.io/projected/721668b5-4594-43bb-82ae-f8fd5e80fd2e-kube-api-access-rzcpr\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.256161 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.253291 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8cgn\" (UniqueName: \"kubernetes.io/projected/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-kube-api-access-h8cgn\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.256161 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.253289 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wdt66\" (UniqueName: \"kubernetes.io/projected/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-kube-api-access-wdt66\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.256161 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.253362 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjbdf\" (UniqueName: \"kubernetes.io/projected/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-kube-api-access-hjbdf\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.256161 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.253372 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8z54h\" (UniqueName: \"kubernetes.io/projected/827a07c8-e408-4526-a58b-4f7982a7dcaa-kube-api-access-8z54h\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.257234 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257070 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cb42l\" (UniqueName: \"kubernetes.io/projected/4d1f10bd-b2d0-486d-9caa-21b650348f93-kube-api-access-cb42l\") pod \"network-metrics-daemon-6dpdv\" (UID: \"4d1f10bd-b2d0-486d-9caa-21b650348f93\") " pod="openshift-multus/network-metrics-daemon-6dpdv" Oct 30 23:54:39.257234 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257078 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b787\" (UniqueName: \"kubernetes.io/projected/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-kube-api-access-9b787\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.257234 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257112 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nd8n\" (UniqueName: \"kubernetes.io/projected/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-kube-api-access-9nd8n\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.257234 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257140 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7ctp6\" (UniqueName: \"kubernetes.io/projected/8411675f-2736-483b-8a74-a15c12287ea1-kube-api-access-7ctp6\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.257418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pls9z\" (UniqueName: \"kubernetes.io/projected/46826886-bbcb-4f1e-ae2a-94f7feb26d76-kube-api-access-pls9z\") pod \"kube-state-metrics-5dc8d7d89b-bnpd7\" (UID: \"46826886-bbcb-4f1e-ae2a-94f7feb26d76\") " pod="openshift-monitoring/kube-state-metrics-5dc8d7d89b-bnpd7" Oct 30 23:54:39.257418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257346 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"alertmanager-main-0\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:39.257418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257346 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tkvgs\" (UniqueName: \"kubernetes.io/projected/fa89d9ae-29cc-4664-ae9d-ca28b785c7bd-kube-api-access-tkvgs\") pod \"network-check-target-8nvcl\" (UID: \"fa89d9ae-29cc-4664-ae9d-ca28b785c7bd\") " pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:39.257705 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.257688 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd7df\" (UniqueName: \"kubernetes.io/projected/32b07947-679b-40bb-849d-10815894262d-kube-api-access-jd7df\") pod \"ingress-canary-5sqx2\" (UID: \"32b07947-679b-40bb-849d-10815894262d\") " pod="openshift-ingress-canary/ingress-canary-5sqx2" Oct 30 23:54:39.260820 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.260772 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58qmb\" (UniqueName: \"kubernetes.io/projected/b71f4919-41b2-469c-af26-b016813419ec-kube-api-access-58qmb\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.266750 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266723 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.266865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266754 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.266865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266805 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.266865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266827 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-sys\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266833 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266882 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysconfig\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266867 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-log-socket\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266895 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-registration-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266902 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266959 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266983 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266997 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-modprobe-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267005 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-multus-certs\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267009 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.267020 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.266985 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d-hosts-file\") pod \"node-resolver-q7jkq\" (UID: \"78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d\") " pod="openshift-dns/node-resolver-q7jkq" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267044 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/721668b5-4594-43bb-82ae-f8fd5e80fd2e-host-slash\") pod \"iptables-alerter-dx424\" (UID: \"721668b5-4594-43bb-82ae-f8fd5e80fd2e\") " pod="openshift-network-operator/iptables-alerter-dx424" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267046 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267072 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-systemd\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267075 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-device-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267111 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267080 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267132 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"console-588d4889bc-vs299\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267157 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267198 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267218 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267239 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267256 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-run\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267263 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267270 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267281 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-cnibin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267286 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267291 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-conf-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267323 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-system-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267315 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-root\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267338 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267344 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-slash\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267361 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267361 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-d\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267377 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267409 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267423 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-systemd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267434 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267437 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-bin\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267444 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-kubelet\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267451 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-cnibin\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267459 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267472 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-kubernetes\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-sys\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.267479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267498 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267507 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-var-lib-kubelet\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267520 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267527 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-bin\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267546 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267550 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-netns\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267556 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267568 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267584 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267589 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267598 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-host\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267604 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267623 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-dbus\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267635 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-cni-netd\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267641 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267660 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267673 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267690 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267704 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267721 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267724 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/827a07c8-e408-4526-a58b-4f7982a7dcaa-host\") pod \"node-ca-4qqrz\" (UID: \"827a07c8-e408-4526-a58b-4f7982a7dcaa\") " pod="openshift-image-registry/node-ca-4qqrz" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267732 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-etc-kubernetes\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267733 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-var-lib-cni-multus\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267746 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267765 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/d775e8d4-9d4a-4779-a5c5-4442e8d3a306-node-exporter-wtmp\") pod \"node-exporter-q2kgm\" (UID: \"d775e8d4-9d4a-4779-a5c5-4442e8d3a306\") " pod="openshift-monitoring/node-exporter-q2kgm" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267772 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267808 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267824 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267829 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/58a99b61-5c4d-41c8-9b7a-172aa3edc72b-crio-socket\") pod \"insights-runtime-extractor-vscwx\" (UID: \"58a99b61-5c4d-41c8-9b7a-172aa3edc72b\") " pod="openshift-insights/insights-runtime-extractor-vscwx" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267839 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267806 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-socket-dir-parent\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267847 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-etc-sysctl-conf\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267839 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-hostroot\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267858 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267866 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-sys-fs\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267879 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-etc-selinux\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267899 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-kubelet\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267908 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-host-run-k8s-cni-cncf-io\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267910 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267884 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-system-cni-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267938 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-node-log\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267945 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267967 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267981 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.267998 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268017 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268032 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268042 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-systemd-units\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268053 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268060 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/4285655f-2bda-4d2b-87f5-b9506c3fdb51-kubelet-config\") pod \"global-pull-secret-syncer-45ssd\" (UID: \"4285655f-2bda-4d2b-87f5-b9506c3fdb51\") " pod="kube-system/global-pull-secret-syncer-45ssd" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268067 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/b71f4919-41b2-469c-af26-b016813419ec-lib-modules\") pod \"tuned-bsxgt\" (UID: \"b71f4919-41b2-469c-af26-b016813419ec\") " pod="openshift-cluster-node-tuning-operator/tuned-bsxgt" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268082 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-host-run-netns\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268088 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-var-lib-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268097 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-etc-openvswitch\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268083 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268114 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae-run-ovn\") pod \"ovnkube-node-lqv9d\" (UID: \"e7bfb22a-c8c3-4051-8f4b-8a2527fc44ae\") " pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268124 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268142 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268089 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/8411675f-2736-483b-8a74-a15c12287ea1-tuning-conf-dir\") pod \"multus-additional-cni-plugins-n72w7\" (UID: \"8411675f-2736-483b-8a74-a15c12287ea1\") " pod="openshift-multus/multus-additional-cni-plugins-n72w7" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268202 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/e88e13a1-2eda-4ec4-968f-0bada5cf17d9-socket-dir\") pod \"aws-ebs-csi-driver-node-7xdzp\" (UID: \"e88e13a1-2eda-4ec4-968f-0bada5cf17d9\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7xdzp" Oct 30 23:54:39.268675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:39.268241 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/44459cea-2fa8-44fb-ad9f-9aeafcdae6df-multus-cni-dir\") pod \"multus-rblpn\" (UID: \"44459cea-2fa8-44fb-ad9f-9aeafcdae6df\") " pod="openshift-multus/multus-rblpn" Oct 30 23:54:40.937666 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:40.937636 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:41.588355 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:41.588328 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-dw7gl" Oct 30 23:54:44.080399 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:44.080365 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:44.080857 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:44.080559 6234 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness" Oct 30 23:54:47.852664 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:47.852589 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:54:48.019029 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:48.018993 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:54:52.469214 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469177 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:52.469686 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469521 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="alertmanager" containerID="cri-o://d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" gracePeriod=120 Oct 30 23:54:52.469686 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469577 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-web" containerID="cri-o://51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" gracePeriod=120 Oct 30 23:54:52.469686 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469623 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="prom-label-proxy" containerID="cri-o://0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" gracePeriod=120 Oct 30 23:54:52.469686 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469585 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-metric" containerID="cri-o://526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" gracePeriod=120 Oct 30 23:54:52.469686 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469628 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="config-reloader" containerID="cri-o://9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" gracePeriod=120 Oct 30 23:54:52.469955 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.469639 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/alertmanager-main-0" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy" containerID="cri-o://396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" gracePeriod=120 Oct 30 23:54:52.711705 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.711678 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:52.772016 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.771885 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772019 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772099 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772150 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772187 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772408 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772222 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772408 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772263 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772408 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772326 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772408 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772358 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772408 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772390 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772680 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772421 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772680 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772447 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.772680 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772472 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") pod \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\" (UID: \"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02\") " Oct 30 23:54:52.773039 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.772730 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "alertmanager-main-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:54:52.775339 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.775064 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.775425 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.775755 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.776276 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "secret-alertmanager-main-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.776552 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.776661 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "alertmanager-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:54:52.778335 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.776975 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "kube-api-access-lsh47". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:54:52.778671 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.778426 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.778671 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.778658 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "config-volume". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.778956 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.778920 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "secret-alertmanager-kube-rbac-proxy-metric". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.780292 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.780165 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "cluster-tls-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.786716 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.786681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config" pod "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" (UID: "ab2ff55d-f7e3-4611-ac17-a4c320b2fb02"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:54:52.873966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873910 6234 reconciler_common.go:299] "Volume detached for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-main-db\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.873966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873955 6234 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-out\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.873966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873967 6234 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-web-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.873966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873977 6234 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-metric\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.873966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873987 6234 reconciler_common.go:299] "Volume detached for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.873997 6234 reconciler_common.go:299] "Volume detached for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-alertmanager-trusted-ca-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874006 6234 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-main-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874015 6234 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874023 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsh47\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-kube-api-access-lsh47\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874032 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-metrics-client-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874040 6234 reconciler_common.go:299] "Volume detached for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-secret-alertmanager-kube-rbac-proxy-web\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874049 6234 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-tls-assets\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:52.874265 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:52.874057 6234 reconciler_common.go:299] "Volume detached for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02-cluster-tls-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:54:53.190653 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190612 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" exitCode=0 Oct 30 23:54:53.190653 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190648 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" exitCode=0 Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190660 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" exitCode=0 Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190668 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" exitCode=0 Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190678 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" exitCode=0 Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190687 6234 generic.go:356] "Generic (PLEG): container finished" podID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" exitCode=0 Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190694 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190732 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190764 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190809 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190812 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190836 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190860 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190884 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} Oct 30 23:54:53.190908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.190906 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"ab2ff55d-f7e3-4611-ac17-a4c320b2fb02","Type":"ContainerDied","Data":"f1d9c92f87b221ae6448aa10ffb6a587a2be53b5e62833d4354e231cea180ec9"} Oct 30 23:54:53.199296 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.199278 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.206757 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.206736 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.213591 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.213548 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.215030 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.215006 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:53.220608 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.220584 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:53.221643 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.221624 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.228479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.228459 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.235382 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.235368 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.242220 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.242201 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.242498 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.242476 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.242574 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.242511 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.242574 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.242554 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.242871 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.242848 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.242919 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.242883 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.242919 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.242902 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.243139 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.243121 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.243180 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243146 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.243180 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243160 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.243381 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.243364 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.243436 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243389 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.243436 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243410 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.243660 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.243644 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.243701 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243665 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.243701 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243678 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.243937 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.243921 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.243977 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243941 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.243977 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.243953 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.244209 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:54:53.244192 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.244269 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244216 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.244269 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244236 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.244461 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244438 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.244505 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244463 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.244666 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244648 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.244710 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244668 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.244923 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244905 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.245000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.244924 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.245166 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245150 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.245211 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245167 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.245373 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245358 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.245415 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245373 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.245587 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245571 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.245653 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245588 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.245827 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245811 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.245882 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.245828 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.246050 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246033 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.246118 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246051 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.246260 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246240 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.246304 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246261 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.246475 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246456 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.246519 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246475 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.246671 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246653 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.246737 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246672 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.246934 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246919 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.246978 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.246934 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.247129 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.247113 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.247183 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.247129 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.248883 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.248913 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249230 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249251 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249514 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.249772 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249531 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.250053 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249923 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.250053 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.249941 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.250222 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.250203 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.250222 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.250221 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.250725 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.250703 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.250820 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.250727 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.251022 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251001 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.251022 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251020 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.251280 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251260 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.251334 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251281 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.251508 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251492 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.251549 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251509 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.251802 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251767 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.251802 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.251799 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.253079 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253061 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:53.253315 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253303 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-web" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253317 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-web" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253326 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="config-reloader" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253331 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="config-reloader" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253340 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="alertmanager" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253344 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="alertmanager" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253355 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="prom-label-proxy" Oct 30 23:54:53.253357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253360 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="prom-label-proxy" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253366 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-metric" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253371 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-metric" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253377 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253381 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253394 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="init-config-reloader" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253398 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="init-config-reloader" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253435 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253443 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="init-config-reloader" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253449 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-web" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253454 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="config-reloader" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253459 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="alertmanager" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253464 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="prom-label-proxy" Oct 30 23:54:53.253551 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.253468 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" containerName="kube-rbac-proxy-metric" Oct 30 23:54:53.254556 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.254538 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.254556 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.254555 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.254921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.254899 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.254921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.254918 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.255185 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255162 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.255185 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255183 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.255452 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255432 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.255521 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255454 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.255712 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255693 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.255758 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255713 6234 scope.go:122] "RemoveContainer" containerID="0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a" Oct 30 23:54:53.255939 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255922 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a"} err="failed to get container status \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": rpc error: code = NotFound desc = could not find container \"0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a\": container with ID starting with 0689d58c3d34404350ae5d21beb5b60cf96af0af49319225f7088e384f95778a not found: ID does not exist" Oct 30 23:54:53.255992 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.255940 6234 scope.go:122] "RemoveContainer" containerID="526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7" Oct 30 23:54:53.256190 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256172 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7"} err="failed to get container status \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": rpc error: code = NotFound desc = could not find container \"526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7\": container with ID starting with 526ea28c4ea0d02414a2ade98121a12443e73f630d878eda2ba9cf2f9c61f1b7 not found: ID does not exist" Oct 30 23:54:53.256242 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256192 6234 scope.go:122] "RemoveContainer" containerID="396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985" Oct 30 23:54:53.256439 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256424 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985"} err="failed to get container status \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": rpc error: code = NotFound desc = could not find container \"396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985\": container with ID starting with 396cf3a6af16b75a271d049156e8fd8ce8e10ee1a1573cf9bc86c61d0e590985 not found: ID does not exist" Oct 30 23:54:53.256489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256439 6234 scope.go:122] "RemoveContainer" containerID="51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131" Oct 30 23:54:53.256640 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256622 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131"} err="failed to get container status \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": rpc error: code = NotFound desc = could not find container \"51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131\": container with ID starting with 51ff90b89843902660c9e7fa407d8c0653219ca9ebf7c4c24c60a545e5861131 not found: ID does not exist" Oct 30 23:54:53.256689 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256640 6234 scope.go:122] "RemoveContainer" containerID="9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209" Oct 30 23:54:53.256875 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256857 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209"} err="failed to get container status \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": rpc error: code = NotFound desc = could not find container \"9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209\": container with ID starting with 9f59cf213aa736d5d13725757a343b81fc139a2002ca88b93d16b0d1a7ec3209 not found: ID does not exist" Oct 30 23:54:53.256930 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.256876 6234 scope.go:122] "RemoveContainer" containerID="d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529" Oct 30 23:54:53.257121 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.257104 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529"} err="failed to get container status \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": rpc error: code = NotFound desc = could not find container \"d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529\": container with ID starting with d208ac46c429af9cefc0d8e751ca3efec1da5b839a703ca6a498150b2cf87529 not found: ID does not exist" Oct 30 23:54:53.257121 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.257121 6234 scope.go:122] "RemoveContainer" containerID="22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8" Oct 30 23:54:53.257337 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.257321 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8"} err="failed to get container status \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": rpc error: code = NotFound desc = could not find container \"22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8\": container with ID starting with 22bc29bd63c1b7dc5675a63b7d56800d7b06c35e0a61a10ee883364efff8c1f8 not found: ID does not exist" Oct 30 23:54:53.257879 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.257866 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.264175 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.263977 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\"" Oct 30 23:54:53.264175 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.264163 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\"" Oct 30 23:54:53.264281 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.264192 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\"" Oct 30 23:54:53.264281 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.264231 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\"" Oct 30 23:54:53.265661 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.265640 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\"" Oct 30 23:54:53.265661 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.265657 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\"" Oct 30 23:54:53.265808 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.265675 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\"" Oct 30 23:54:53.268146 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.268131 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-wcgjs\"" Oct 30 23:54:53.268321 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.268305 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\"" Oct 30 23:54:53.270859 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.270841 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\"" Oct 30 23:54:53.278496 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.278470 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:53.379116 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379078 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379116 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379116 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379324 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379195 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379324 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379217 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-web-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379324 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379235 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379324 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379307 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379449 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379340 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-volume\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379449 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379383 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379449 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379404 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379449 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379434 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vc6jn\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-kube-api-access-vc6jn\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379564 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379470 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379564 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379507 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-out\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.379564 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.379523 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480714 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480744 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vc6jn\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-kube-api-access-vc6jn\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480763 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480799 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-out\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480822 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.480863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480845 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480878 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480924 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480947 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-web-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.480974 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.481016 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.481046 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-volume\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.481089 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.481530 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.481164 6234 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory" Oct 30 23:54:53.483617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.482266 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.483617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.482399 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.483617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.483417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.483885 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.483864 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-out\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.484276 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.484219 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.484506 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.484485 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.484593 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.484489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.486192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.486163 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-web-config\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.486355 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.486334 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-config-volume\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.486493 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.486477 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-tls-assets\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.486550 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.486479 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.486726 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.486708 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.493995 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.493974 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vc6jn\" (UniqueName: \"kubernetes.io/projected/18dfcc6b-111a-4d7b-ad7e-8bd92264af58-kube-api-access-vc6jn\") pod \"alertmanager-main-0\" (UID: \"18dfcc6b-111a-4d7b-ad7e-8bd92264af58\") " pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.585548 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.585498 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0" Oct 30 23:54:53.873199 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:53.873128 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"] Oct 30 23:54:53.876774 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:53.876746 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18dfcc6b_111a_4d7b_ad7e_8bd92264af58.slice/crio-98c32a923d0c9a131cdaac7f618119555e5de621a4042c82f7c152ae219ef391 WatchSource:0}: Error finding container 98c32a923d0c9a131cdaac7f618119555e5de621a4042c82f7c152ae219ef391: Status 404 returned error can't find the container with id 98c32a923d0c9a131cdaac7f618119555e5de621a4042c82f7c152ae219ef391 Oct 30 23:54:54.079432 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:54.079394 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab2ff55d-f7e3-4611-ac17-a4c320b2fb02" path="/var/lib/kubelet/pods/ab2ff55d-f7e3-4611-ac17-a4c320b2fb02/volumes" Oct 30 23:54:54.197365 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:54.197328 6234 generic.go:356] "Generic (PLEG): container finished" podID="18dfcc6b-111a-4d7b-ad7e-8bd92264af58" containerID="ce1785a05b2a800a5428b737c96c972ec3ef11187410d5d91f607b594912a46e" exitCode=0 Oct 30 23:54:54.197538 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:54.197368 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerDied","Data":"ce1785a05b2a800a5428b737c96c972ec3ef11187410d5d91f607b594912a46e"} Oct 30 23:54:54.197538 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:54.197432 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"98c32a923d0c9a131cdaac7f618119555e5de621a4042c82f7c152ae219ef391"} Oct 30 23:54:55.204846 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.204815 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"312596cd8867e3579986829ce9378b265ce56b7d31b152ae21fe6955821dfa95"} Oct 30 23:54:55.205243 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.204901 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"66afc5fdd9ab9b20a48b1435a8e82a593d5b8016621d7413a29fb471bb9dfbee"} Oct 30 23:54:55.205243 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.204930 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"ccb028e53d89f3d8b4c6fceda27b51fb7b252b628066205452e3c64f7d16e265"} Oct 30 23:54:55.205243 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.204959 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"98e87f04e8e2427a87d024f9b9ea7f53f39127218739bb960085bd3b3f0e8827"} Oct 30 23:54:55.205243 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.204982 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"d140b44acc8e7855d3bb3f0aaac797469230040ac97edc1140499fe2e7d00a5d"} Oct 30 23:54:55.205243 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.205003 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"18dfcc6b-111a-4d7b-ad7e-8bd92264af58","Type":"ContainerStarted","Data":"35cc46584048b6d346dd486bb22fb128d4cc4f8d22eb4ce06c7def423b3030ea"} Oct 30 23:54:55.242000 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.241942 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=2.2233316260000002 podStartE2EDuration="2.241928705s" totalImagesPullingTime="0s" totalInitContainerRuntime="18.597079ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:54:55.239616591 +0000 UTC m=+17.272712698" watchObservedRunningTime="2026-10-30 23:54:55.241928705 +0000 UTC m=+17.275024775" Oct 30 23:54:55.698420 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:55.698394 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-lqv9d" Oct 30 23:54:56.490084 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.490050 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-6c4d947c48-bmwpb"] Oct 30 23:54:56.494169 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.494148 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.498692 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.498673 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\"" Oct 30 23:54:56.498692 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.498680 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" Oct 30 23:54:56.498692 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.498680 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" Oct 30 23:54:56.498911 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.498762 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" Oct 30 23:54:56.500698 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.500619 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-mthwl\"" Oct 30 23:54:56.500954 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.500936 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" Oct 30 23:54:56.507798 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.507744 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" Oct 30 23:54:56.508775 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.508756 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4d947c48-bmwpb"] Oct 30 23:54:56.609085 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609052 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609085 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609088 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609312 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609113 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-serving-certs-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609312 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609273 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-federate-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609331 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvpc5\" (UniqueName: \"kubernetes.io/projected/9c881400-6424-435d-9c12-2fc6ba5d25f4-kube-api-access-wvpc5\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609402 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609496 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609428 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-metrics-client-ca\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.609496 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.609452 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.710568 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.710526 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.710568 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.710572 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.710824 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.710625 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-serving-certs-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.711135 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.711088 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-federate-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.711223 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.711177 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvpc5\" (UniqueName: \"kubernetes.io/projected/9c881400-6424-435d-9c12-2fc6ba5d25f4-kube-api-access-wvpc5\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.711273 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.711241 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.711329 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.711273 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-metrics-client-ca\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.711329 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.711311 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.712527 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.712112 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-serving-certs-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.712527 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.712116 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-trusted-ca-bundle\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.712995 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.712968 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9c881400-6424-435d-9c12-2fc6ba5d25f4-metrics-client-ca\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.714154 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.714127 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.714392 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.714369 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-secret-telemeter-client\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.715692 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.715662 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-telemeter-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.717546 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.717525 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/9c881400-6424-435d-9c12-2fc6ba5d25f4-federate-client-tls\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.725203 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.725184 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvpc5\" (UniqueName: \"kubernetes.io/projected/9c881400-6424-435d-9c12-2fc6ba5d25f4-kube-api-access-wvpc5\") pod \"telemeter-client-6c4d947c48-bmwpb\" (UID: \"9c881400-6424-435d-9c12-2fc6ba5d25f4\") " pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:56.812950 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:56.812868 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" Oct 30 23:54:57.098833 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:57.098741 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-6c4d947c48-bmwpb"] Oct 30 23:54:57.101799 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:54:57.101758 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c881400_6424_435d_9c12_2fc6ba5d25f4.slice/crio-21edfb6de30e623f794bcadd40fdf563e72d1e1acf65c6e77f06ed4853611779 WatchSource:0}: Error finding container 21edfb6de30e623f794bcadd40fdf563e72d1e1acf65c6e77f06ed4853611779: Status 404 returned error can't find the container with id 21edfb6de30e623f794bcadd40fdf563e72d1e1acf65c6e77f06ed4853611779 Oct 30 23:54:57.103743 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:57.103728 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 30 23:54:57.214920 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:57.214882 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" event={"ID":"9c881400-6424-435d-9c12-2fc6ba5d25f4","Type":"ContainerStarted","Data":"21edfb6de30e623f794bcadd40fdf563e72d1e1acf65c6e77f06ed4853611779"} Oct 30 23:54:59.224975 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:59.224947 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" event={"ID":"9c881400-6424-435d-9c12-2fc6ba5d25f4","Type":"ContainerStarted","Data":"a1965371c7859869a667872265135f55af7d8fc734684b46621cdd2f3641c730"} Oct 30 23:54:59.225269 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:54:59.224999 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" event={"ID":"9c881400-6424-435d-9c12-2fc6ba5d25f4","Type":"ContainerStarted","Data":"e9a72af7e14ab80d8c79d5d961787c8a3a6be498f285264db100bd810cee7417"} Oct 30 23:55:00.230383 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.230347 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" event={"ID":"9c881400-6424-435d-9c12-2fc6ba5d25f4","Type":"ContainerStarted","Data":"4d6acef9238203a5e9ec40efc5069eb3109cf457aff2533419279fcca436149a"} Oct 30 23:55:00.262639 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.262582 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-6c4d947c48-bmwpb" podStartSLOduration=2.312684067 podStartE2EDuration="4.262566837s" totalImagesPullingTime="1.94988277s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:54:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:00.262079205 +0000 UTC m=+22.295175275" watchObservedRunningTime="2026-10-30 23:55:00.262566837 +0000 UTC m=+22.295662911" Oct 30 23:55:00.355804 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.355715 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:00.359131 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.359113 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.372343 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.372318 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:00.443057 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443016 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443242 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443070 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443242 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443139 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443242 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443163 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443242 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443177 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zmrk\" (UniqueName: \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443363 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443317 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443398 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443355 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.443429 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.443407 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.544855 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.544812 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.544855 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.544857 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zmrk\" (UniqueName: \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.544911 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.544940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.544971 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545042 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545071 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545139 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545102 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545495 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545469 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545716 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545693 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.545716 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.545706 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.546030 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.546003 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.546077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.546046 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.547567 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.547548 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.547651 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.547632 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.558455 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.558426 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zmrk\" (UniqueName: \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\") pod \"console-5b6bdc5489-nb26l\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.670563 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.670473 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:00.949175 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:00.949144 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:00.952403 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:55:00.952375 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57713365_2cf7_4a7a_b926_93338b8629e2.slice/crio-06cbf93182926751101b28373f13bc0cb9b66a44b7c0b5fd6839c1bc46674094 WatchSource:0}: Error finding container 06cbf93182926751101b28373f13bc0cb9b66a44b7c0b5fd6839c1bc46674094: Status 404 returned error can't find the container with id 06cbf93182926751101b28373f13bc0cb9b66a44b7c0b5fd6839c1bc46674094 Oct 30 23:55:01.235464 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:01.235371 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bdc5489-nb26l" event={"ID":"57713365-2cf7-4a7a-b926-93338b8629e2","Type":"ContainerStarted","Data":"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03"} Oct 30 23:55:01.235464 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:01.235425 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bdc5489-nb26l" event={"ID":"57713365-2cf7-4a7a-b926-93338b8629e2","Type":"ContainerStarted","Data":"06cbf93182926751101b28373f13bc0cb9b66a44b7c0b5fd6839c1bc46674094"} Oct 30 23:55:01.258101 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:01.258047 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-5b6bdc5489-nb26l" podStartSLOduration=1.258034178 podStartE2EDuration="1.258034178s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:01.257938923 +0000 UTC m=+23.291034990" watchObservedRunningTime="2026-10-30 23:55:01.258034178 +0000 UTC m=+23.291130229" Oct 30 23:55:01.684105 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:01.684023 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-dbh42" Oct 30 23:55:03.159661 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:55:03.159626 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:55:04.078658 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.078588 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-b4d4f6d76-jd8lq" podUID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" containerName="console" containerID="cri-o://5a2c9c5812d5c887a8f698d60bec19814a03f491478d7b59b7cbaa7844463ed3" gracePeriod=15 Oct 30 23:55:04.084212 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.084174 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-6799bcd5d4-r6r54" podUID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" containerName="console" containerID="cri-o://cc484ead940bff20eeccc55c29b23bbf29b19187d163830003105fddb88f826b" gracePeriod=15 Oct 30 23:55:04.249739 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.249712 6234 generic.go:356] "Generic (PLEG): container finished" podID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" containerID="5a2c9c5812d5c887a8f698d60bec19814a03f491478d7b59b7cbaa7844463ed3" exitCode=0 Oct 30 23:55:04.250085 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.249805 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d4f6d76-jd8lq" event={"ID":"398381a0-d000-45eb-8c18-a1f6b2f2ed60","Type":"ContainerDied","Data":"5a2c9c5812d5c887a8f698d60bec19814a03f491478d7b59b7cbaa7844463ed3"} Oct 30 23:55:04.251436 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.251412 6234 generic.go:356] "Generic (PLEG): container finished" podID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" containerID="cc484ead940bff20eeccc55c29b23bbf29b19187d163830003105fddb88f826b" exitCode=0 Oct 30 23:55:04.251555 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.251442 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6799bcd5d4-r6r54" event={"ID":"3fbfc93b-8923-4b81-9f55-c4578c1c25f3","Type":"ContainerDied","Data":"cc484ead940bff20eeccc55c29b23bbf29b19187d163830003105fddb88f826b"} Oct 30 23:55:04.357102 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.357076 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:55:04.359661 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.359642 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:55:04.485602 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485563 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.485602 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485611 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.485857 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485643 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.485857 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485684 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.485857 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485732 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.485857 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485812 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485868 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485898 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485945 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.485980 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486014 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.486063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486042 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.486366 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486075 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.486366 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486083 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:04.486366 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486106 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") pod \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\" (UID: \"398381a0-d000-45eb-8c18-a1f6b2f2ed60\") " Oct 30 23:55:04.486366 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486128 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.486366 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486140 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") pod \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\" (UID: \"3fbfc93b-8923-4b81-9f55-c4578c1c25f3\") " Oct 30 23:55:04.486617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486395 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-oauth-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.486617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486416 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-tmp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.486617 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486135 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.486830 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486616 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:04.486830 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486583 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.487073 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486965 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.487207 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.487091 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.487207 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.486498 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.487207 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.487151 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:04.488718 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.488673 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:04.488868 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.488732 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "kube-api-access-f6f7l". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:04.488950 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.488877 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:04.489111 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.489091 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "kube-api-access-wqd5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:04.489304 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.489282 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert" pod "398381a0-d000-45eb-8c18-a1f6b2f2ed60" (UID: "398381a0-d000-45eb-8c18-a1f6b2f2ed60"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:04.489344 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.489309 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config" pod "3fbfc93b-8923-4b81-9f55-c4578c1c25f3" (UID: "3fbfc93b-8923-4b81-9f55-c4578c1c25f3"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587129 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587162 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqd5r\" (UniqueName: \"kubernetes.io/projected/398381a0-d000-45eb-8c18-a1f6b2f2ed60-kube-api-access-wqd5r\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587174 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587184 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-oauth-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587193 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-oauth-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587201 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-console-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587210 6234 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-trusted-ca-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587215 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587219 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/398381a0-d000-45eb-8c18-a1f6b2f2ed60-tmp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587559 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587229 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587559 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587239 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6f7l\" (UniqueName: \"kubernetes.io/projected/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-kube-api-access-f6f7l\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587559 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587248 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/398381a0-d000-45eb-8c18-a1f6b2f2ed60-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587559 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587257 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/398381a0-d000-45eb-8c18-a1f6b2f2ed60-console-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:04.587559 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:04.587265 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3fbfc93b-8923-4b81-9f55-c4578c1c25f3-oauth-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:05.256516 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.256481 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-b4d4f6d76-jd8lq" Oct 30 23:55:05.256516 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.256492 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-b4d4f6d76-jd8lq" event={"ID":"398381a0-d000-45eb-8c18-a1f6b2f2ed60","Type":"ContainerDied","Data":"52e0fa6449d30a4e6b770409bf5dc047e039f310271242bf24c4757f86152fb8"} Oct 30 23:55:05.257049 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.256560 6234 scope.go:122] "RemoveContainer" containerID="5a2c9c5812d5c887a8f698d60bec19814a03f491478d7b59b7cbaa7844463ed3" Oct 30 23:55:05.261474 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.261444 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-6799bcd5d4-r6r54" event={"ID":"3fbfc93b-8923-4b81-9f55-c4578c1c25f3","Type":"ContainerDied","Data":"e97f7b1e31c71fefa184ea8712a0ac2e224f342a1973b3f0141461220c7ccb11"} Oct 30 23:55:05.261571 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.261557 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-6799bcd5d4-r6r54" Oct 30 23:55:05.269465 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.269445 6234 scope.go:122] "RemoveContainer" containerID="cc484ead940bff20eeccc55c29b23bbf29b19187d163830003105fddb88f826b" Oct 30 23:55:05.285753 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.285730 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:55:05.290202 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.290181 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-b4d4f6d76-jd8lq"] Oct 30 23:55:05.305145 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.305119 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:55:05.315919 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:05.315891 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-6799bcd5d4-r6r54"] Oct 30 23:55:06.078725 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:06.078690 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" path="/var/lib/kubelet/pods/398381a0-d000-45eb-8c18-a1f6b2f2ed60/volumes" Oct 30 23:55:06.079118 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:06.079105 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" path="/var/lib/kubelet/pods/3fbfc93b-8923-4b81-9f55-c4578c1c25f3/volumes" Oct 30 23:55:08.021957 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.021932 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-8nvcl" Oct 30 23:55:08.886800 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.886761 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:08.923480 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923446 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/console-75bdffcc5c-glhxn"] Oct 30 23:55:08.923776 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923763 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" containerName="console" Oct 30 23:55:08.923842 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923792 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" containerName="console" Oct 30 23:55:08.923842 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923804 6234 cpu_manager.go:414] "containerMap: removing container" podUID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" containerName="console" Oct 30 23:55:08.923842 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923809 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" containerName="console" Oct 30 23:55:08.923929 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923862 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3fbfc93b-8923-4b81-9f55-c4578c1c25f3" containerName="console" Oct 30 23:55:08.923929 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.923870 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="398381a0-d000-45eb-8c18-a1f6b2f2ed60" containerName="console" Oct 30 23:55:08.962142 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.962106 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bdffcc5c-glhxn"] Oct 30 23:55:08.962282 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:08.962221 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.027863 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.027829 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-oauth-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.027873 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-oauth-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.027896 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a197b72-71ae-4bc5-9736-12979d594ab6-tmp\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.027969 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-console-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.028026 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-trusted-ca-bundle\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.028067 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-service-ca\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.028113 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nkthz\" (UniqueName: \"kubernetes.io/projected/3a197b72-71ae-4bc5-9736-12979d594ab6-kube-api-access-nkthz\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.028236 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.028129 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129097 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129065 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-console-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129097 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129101 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-trusted-ca-bundle\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129126 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-service-ca\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129165 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nkthz\" (UniqueName: \"kubernetes.io/projected/3a197b72-71ae-4bc5-9736-12979d594ab6-kube-api-access-nkthz\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129194 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129259 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-oauth-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129299 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-oauth-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129557 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129330 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a197b72-71ae-4bc5-9736-12979d594ab6-tmp\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.129816 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129772 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3a197b72-71ae-4bc5-9736-12979d594ab6-tmp\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.130046 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.129993 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-service-ca\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.130127 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.130005 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-oauth-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.130208 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.130192 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-trusted-ca-bundle\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.130274 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.130253 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3a197b72-71ae-4bc5-9736-12979d594ab6-console-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.131966 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.131937 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-oauth-config\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.132054 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.131976 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3a197b72-71ae-4bc5-9736-12979d594ab6-console-serving-cert\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.142717 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.142677 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nkthz\" (UniqueName: \"kubernetes.io/projected/3a197b72-71ae-4bc5-9736-12979d594ab6-kube-api-access-nkthz\") pod \"console-75bdffcc5c-glhxn\" (UID: \"3a197b72-71ae-4bc5-9736-12979d594ab6\") " pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.284146 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.284112 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:09.592666 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:09.592644 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-75bdffcc5c-glhxn"] Oct 30 23:55:09.595224 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:55:09.595193 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a197b72_71ae_4bc5_9736_12979d594ab6.slice/crio-f614ee21f141a153ab77a067116a1d1805723f176cd20a3681ae6403b7e3cd91 WatchSource:0}: Error finding container f614ee21f141a153ab77a067116a1d1805723f176cd20a3681ae6403b7e3cd91: Status 404 returned error can't find the container with id f614ee21f141a153ab77a067116a1d1805723f176cd20a3681ae6403b7e3cd91 Oct 30 23:55:10.283156 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:10.283126 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bdffcc5c-glhxn" event={"ID":"3a197b72-71ae-4bc5-9736-12979d594ab6","Type":"ContainerStarted","Data":"d8a3e925fff92e7a1885626cc17307b7ff031284a8bacd36675616b510a85cd6"} Oct 30 23:55:10.283531 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:10.283171 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-75bdffcc5c-glhxn" event={"ID":"3a197b72-71ae-4bc5-9736-12979d594ab6","Type":"ContainerStarted","Data":"f614ee21f141a153ab77a067116a1d1805723f176cd20a3681ae6403b7e3cd91"} Oct 30 23:55:10.306532 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:10.306481 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/console-75bdffcc5c-glhxn" podStartSLOduration=2.306469075 podStartE2EDuration="2.306469075s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:55:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:55:10.305910835 +0000 UTC m=+32.339006910" watchObservedRunningTime="2026-10-30 23:55:10.306469075 +0000 UTC m=+32.339565145" Oct 30 23:55:10.671638 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:10.671554 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:19.284613 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:19.284573 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:19.285127 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:19.284639 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:19.289255 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:19.289233 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:19.318437 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:19.318418 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/console-75bdffcc5c-glhxn" Oct 30 23:55:19.368346 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:19.368320 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:55:33.159531 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:55:33.159490 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:55:33.906418 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:33.906361 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-5b6bdc5489-nb26l" podUID="57713365-2cf7-4a7a-b926-93338b8629e2" containerName="console" containerID="cri-o://9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03" gracePeriod=15 Oct 30 23:55:34.155414 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.155391 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:34.233370 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233336 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\" (UniqueName: \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233382 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233410 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\" (UniqueName: \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233448 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233479 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233518 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233560 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233588 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\") pod \"57713365-2cf7-4a7a-b926-93338b8629e2\" (UID: \"57713365-2cf7-4a7a-b926-93338b8629e2\") " Oct 30 23:55:34.233844 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233624 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:34.234270 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233870 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/57713365-2cf7-4a7a-b926-93338b8629e2-tmp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.234270 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.233997 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:34.234270 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.234016 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:34.234270 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.234150 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:34.234270 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.234167 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:34.235910 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.235880 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:34.236195 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.236176 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:34.236255 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.236194 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk" pod "57713365-2cf7-4a7a-b926-93338b8629e2" (UID: "57713365-2cf7-4a7a-b926-93338b8629e2"). InnerVolumeSpecName "kube-api-access-7zmrk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:34.334479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334427 6234 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-trusted-ca-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334473 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334479 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334486 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334731 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334515 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/57713365-2cf7-4a7a-b926-93338b8629e2-console-oauth-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334731 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334527 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zmrk\" (UniqueName: \"kubernetes.io/projected/57713365-2cf7-4a7a-b926-93338b8629e2-kube-api-access-7zmrk\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334731 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334540 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-oauth-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.334731 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.334552 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/57713365-2cf7-4a7a-b926-93338b8629e2-console-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:34.368616 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.368584 6234 generic.go:356] "Generic (PLEG): container finished" podID="57713365-2cf7-4a7a-b926-93338b8629e2" containerID="9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03" exitCode=0 Oct 30 23:55:34.368768 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.368637 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-5b6bdc5489-nb26l" Oct 30 23:55:34.368768 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.368643 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bdc5489-nb26l" event={"ID":"57713365-2cf7-4a7a-b926-93338b8629e2","Type":"ContainerDied","Data":"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03"} Oct 30 23:55:34.368768 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.368758 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-5b6bdc5489-nb26l" event={"ID":"57713365-2cf7-4a7a-b926-93338b8629e2","Type":"ContainerDied","Data":"06cbf93182926751101b28373f13bc0cb9b66a44b7c0b5fd6839c1bc46674094"} Oct 30 23:55:34.368942 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.368803 6234 scope.go:122] "RemoveContainer" containerID="9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03" Oct 30 23:55:34.377627 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.377611 6234 scope.go:122] "RemoveContainer" containerID="9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03" Oct 30 23:55:34.377903 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:55:34.377886 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03\": container with ID starting with 9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03 not found: ID does not exist" containerID="9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03" Oct 30 23:55:34.377973 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.377919 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03"} err="failed to get container status \"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03\": rpc error: code = NotFound desc = could not find container \"9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03\": container with ID starting with 9c4acd7ba878964945b836ac89dbb4b60e34fca9e5f92cf746ed71045611ab03 not found: ID does not exist" Oct 30 23:55:34.392222 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.392190 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:34.397819 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:34.397774 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-5b6bdc5489-nb26l"] Oct 30 23:55:36.078600 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:36.078568 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57713365-2cf7-4a7a-b926-93338b8629e2" path="/var/lib/kubelet/pods/57713365-2cf7-4a7a-b926-93338b8629e2/volumes" Oct 30 23:55:44.388414 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.388344 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/console-588d4889bc-vs299" podUID="3027d685-5695-4d70-8759-5b28d323af4f" containerName="console" containerID="cri-o://a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7" gracePeriod=15 Oct 30 23:55:44.637062 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.637034 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:55:44.719904 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.719871 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720062 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.719931 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720062 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.719972 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720159 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720061 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720159 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720121 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720261 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720180 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720261 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720225 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720261 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720249 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") pod \"3027d685-5695-4d70-8759-5b28d323af4f\" (UID: \"3027d685-5695-4d70-8759-5b28d323af4f\") " Oct 30 23:55:44.720921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720757 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:44.720921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.720767 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:44.721301 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721032 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:44.721419 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721291 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:55:44.721485 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721446 6234 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-oauth-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.721485 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721478 6234 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.721596 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721495 6234 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/3027d685-5695-4d70-8759-5b28d323af4f-tmp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.722943 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.721730 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:55:44.725340 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.723496 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "kube-api-access-hgw25". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:55:44.725537 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.725489 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:44.726891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.726864 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config" pod "3027d685-5695-4d70-8759-5b28d323af4f" (UID: "3027d685-5695-4d70-8759-5b28d323af4f"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:55:44.821892 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.821848 6234 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-oauth-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.821892 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.821884 6234 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-console-config\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.821892 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.821897 6234 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/3027d685-5695-4d70-8759-5b28d323af4f-console-serving-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.822130 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.821917 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hgw25\" (UniqueName: \"kubernetes.io/projected/3027d685-5695-4d70-8759-5b28d323af4f-kube-api-access-hgw25\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:44.822130 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:44.821931 6234 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/3027d685-5695-4d70-8759-5b28d323af4f-trusted-ca-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:55:45.407918 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.407889 6234 generic.go:356] "Generic (PLEG): container finished" podID="3027d685-5695-4d70-8759-5b28d323af4f" containerID="a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7" exitCode=0 Oct 30 23:55:45.408338 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.407951 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-588d4889bc-vs299" Oct 30 23:55:45.408338 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.407981 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d4889bc-vs299" event={"ID":"3027d685-5695-4d70-8759-5b28d323af4f","Type":"ContainerDied","Data":"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7"} Oct 30 23:55:45.408338 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.408033 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-588d4889bc-vs299" event={"ID":"3027d685-5695-4d70-8759-5b28d323af4f","Type":"ContainerDied","Data":"a7018363aa7f0cc192b27018c78bf6887d8c9f4b82aa00264cbcbb04c472cd3f"} Oct 30 23:55:45.408338 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.408056 6234 scope.go:122] "RemoveContainer" containerID="a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7" Oct 30 23:55:45.416345 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.416326 6234 scope.go:122] "RemoveContainer" containerID="a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7" Oct 30 23:55:45.416626 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:55:45.416606 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7\": container with ID starting with a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7 not found: ID does not exist" containerID="a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7" Oct 30 23:55:45.416690 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.416641 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7"} err="failed to get container status \"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7\": rpc error: code = NotFound desc = could not find container \"a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7\": container with ID starting with a73d4b70a26e1437afa4589a3dd40e9aa22aefcb899cf705077e20be3c9e59d7 not found: ID does not exist" Oct 30 23:55:45.429758 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.429733 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:55:45.435577 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:45.435555 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-588d4889bc-vs299"] Oct 30 23:55:46.078973 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:55:46.078934 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3027d685-5695-4d70-8759-5b28d323af4f" path="/var/lib/kubelet/pods/3027d685-5695-4d70-8759-5b28d323af4f/volumes" Oct 30 23:56:03.159994 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:56:03.159954 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:56:33.159812 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:56:33.159717 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:57:03.160259 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:57:03.160219 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:57:25.049105 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049067 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c"] Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049486 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3027d685-5695-4d70-8759-5b28d323af4f" containerName="console" Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049503 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3027d685-5695-4d70-8759-5b28d323af4f" containerName="console" Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049519 6234 cpu_manager.go:414] "containerMap: removing container" podUID="57713365-2cf7-4a7a-b926-93338b8629e2" containerName="console" Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049526 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57713365-2cf7-4a7a-b926-93338b8629e2" containerName="console" Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049599 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3027d685-5695-4d70-8759-5b28d323af4f" containerName="console" Oct 30 23:57:25.049646 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.049610 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="57713365-2cf7-4a7a-b926-93338b8629e2" containerName="console" Oct 30 23:57:25.052939 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.052908 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.075311 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.073252 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"kube-root-ca.crt\"" Oct 30 23:57:25.075311 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.073328 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-marketplace\"/\"openshift-service-ca.crt\"" Oct 30 23:57:25.075311 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.073422 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c4959aeb960c8e20c88ac2de924a35b8221494-145dcd5a-dockercfg-kmlqf\"" Oct 30 23:57:25.086705 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.086677 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c"] Oct 30 23:57:25.161380 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.161346 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.161380 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.161380 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.161633 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.161409 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6fsq\" (UniqueName: \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.164189 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.164164 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc"] Oct 30 23:57:25.167495 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.167474 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.192493 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.192473 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-marketplace\"/\"c7537007944577f64df34960ace3c69136c416-139ec370-dockercfg-k9g7p\"" Oct 30 23:57:25.193190 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.193166 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc"] Oct 30 23:57:25.262287 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262254 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9tbx4\" (UniqueName: \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.262287 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262294 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.262515 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262395 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.262515 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262429 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.262515 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262457 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.262515 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6fsq\" (UniqueName: \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.262778 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262761 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.262832 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.262805 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.276042 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.276020 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6fsq\" (UniqueName: \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\") pod \"c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.363403 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.363321 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.363403 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.363369 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9tbx4\" (UniqueName: \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.363620 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.363489 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.363712 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.363694 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.363867 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.363850 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.376297 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.376265 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9tbx4\" (UniqueName: \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\") pod \"c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.383082 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.383066 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:25.482505 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.482471 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:25.525856 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.525821 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:57:25.529108 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.529080 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.534587 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.534564 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" Oct 30 23:57:25.534707 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.534683 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" Oct 30 23:57:25.534773 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.534730 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" Oct 30 23:57:25.535561 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.535544 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" Oct 30 23:57:25.535649 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.535610 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" Oct 30 23:57:25.535649 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.535638 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\"" Oct 30 23:57:25.535766 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.535674 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-w2ptn\"" Oct 30 23:57:25.544636 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.544584 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:57:25.565429 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565405 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565538 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565437 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565538 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565524 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565622 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565568 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565622 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565587 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565715 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565700 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.565777 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.565761 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4sxh6\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.661749 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.661682 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c"] Oct 30 23:57:25.664639 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:57:25.664612 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23f642a1_ada6_458a_a8e3_7e5f6014a54f.slice/crio-2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2 WatchSource:0}: Error finding container 2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2: Status 404 returned error can't find the container with id 2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2 Oct 30 23:57:25.666116 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666089 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666180 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666135 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666180 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666163 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666387 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666387 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666218 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666387 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666253 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4sxh6\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.666387 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.666287 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.667374 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.667277 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.668586 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.668569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.668712 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.668695 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.669104 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.669078 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.669160 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.669115 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.679897 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.679880 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.680181 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.680164 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4sxh6\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\") pod \"istiod-openshift-gateway-7876bfcb74-8rgd8\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:25.739763 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.739724 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" event={"ID":"23f642a1-ada6-458a-a8e3-7e5f6014a54f","Type":"ContainerStarted","Data":"2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2"} Oct 30 23:57:25.757211 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.757182 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc"] Oct 30 23:57:25.760178 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:57:25.760152 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbc24195_a25b_4ac9_aa24_7e1c10b0e3cd.slice/crio-a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb WatchSource:0}: Error finding container a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb: Status 404 returned error can't find the container with id a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb Oct 30 23:57:25.848738 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:25.848704 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:26.126333 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:26.126297 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:57:26.129432 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:57:26.129406 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd747c179_cb07_4b21_be46_208342311286.slice/crio-c14976152f8689c379929e9524385bb5616d43206d1de659a2794ef14b4fb3a9 WatchSource:0}: Error finding container c14976152f8689c379929e9524385bb5616d43206d1de659a2794ef14b4fb3a9: Status 404 returned error can't find the container with id c14976152f8689c379929e9524385bb5616d43206d1de659a2794ef14b4fb3a9 Oct 30 23:57:26.745908 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:26.745804 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" event={"ID":"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd","Type":"ContainerStarted","Data":"a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb"} Oct 30 23:57:26.747645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:26.747611 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" event={"ID":"d747c179-cb07-4b21-be46-208342311286","Type":"ContainerStarted","Data":"c14976152f8689c379929e9524385bb5616d43206d1de659a2794ef14b4fb3a9"} Oct 30 23:57:30.055312 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:30.055276 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 23:57:30.055626 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:30.055344 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 23:57:30.771425 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:30.771381 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" event={"ID":"d747c179-cb07-4b21-be46-208342311286","Type":"ContainerStarted","Data":"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37"} Oct 30 23:57:30.771632 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:30.771471 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:30.804199 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:30.804139 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" podStartSLOduration=1.881243261 podStartE2EDuration="5.804122922s" totalImagesPullingTime="3.922879661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:57:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:57:30.801226773 +0000 UTC m=+172.834322866" watchObservedRunningTime="2026-10-30 23:57:30.804122922 +0000 UTC m=+172.837218994" Oct 30 23:57:31.779168 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:31.778832 6234 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7876bfcb74-8rgd8 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 30 23:57:31.779168 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:31.778907 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" podUID="d747c179-cb07-4b21-be46-208342311286" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 30 23:57:32.781320 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:32.781289 6234 generic.go:356] "Generic (PLEG): container finished" podID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerID="6d014341ca62953036b0fb5418d439db23934dc2c739c58911d509fe5eec8ada" exitCode=0 Oct 30 23:57:32.781716 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:32.781368 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" event={"ID":"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd","Type":"ContainerDied","Data":"6d014341ca62953036b0fb5418d439db23934dc2c739c58911d509fe5eec8ada"} Oct 30 23:57:32.783425 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:32.783402 6234 generic.go:356] "Generic (PLEG): container finished" podID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerID="8acd0117ff2e5c35a1febd7918099e99dc504ad54479828438d0267d03e78f6f" exitCode=0 Oct 30 23:57:32.783511 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:32.783480 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" event={"ID":"23f642a1-ada6-458a-a8e3-7e5f6014a54f","Type":"ContainerDied","Data":"8acd0117ff2e5c35a1febd7918099e99dc504ad54479828438d0267d03e78f6f"} Oct 30 23:57:33.159148 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:57:33.159111 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:57:34.777798 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:34.777744 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:57:35.797319 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:35.797288 6234 generic.go:356] "Generic (PLEG): container finished" podID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerID="93e32a5961954270f4162ad988dc1d7187a6510f85ceddbe51c1af679d64814e" exitCode=0 Oct 30 23:57:35.797729 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:35.797376 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" event={"ID":"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd","Type":"ContainerDied","Data":"93e32a5961954270f4162ad988dc1d7187a6510f85ceddbe51c1af679d64814e"} Oct 30 23:57:35.799247 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:35.799205 6234 generic.go:356] "Generic (PLEG): container finished" podID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerID="4a9416dcb67a6e1515223a745b856fa45175e49b41f1aec92c49e7d9e2178c85" exitCode=0 Oct 30 23:57:35.799309 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:35.799287 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" event={"ID":"23f642a1-ada6-458a-a8e3-7e5f6014a54f","Type":"ContainerDied","Data":"4a9416dcb67a6e1515223a745b856fa45175e49b41f1aec92c49e7d9e2178c85"} Oct 30 23:57:42.837345 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:42.837273 6234 generic.go:356] "Generic (PLEG): container finished" podID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerID="33a6f7a932eaa28fd072c4bb3b213d4e51c003929e7239a6bf15dd837bfe7ae9" exitCode=0 Oct 30 23:57:42.837345 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:42.837311 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" event={"ID":"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd","Type":"ContainerDied","Data":"33a6f7a932eaa28fd072c4bb3b213d4e51c003929e7239a6bf15dd837bfe7ae9"} Oct 30 23:57:43.842544 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:43.842507 6234 generic.go:356] "Generic (PLEG): container finished" podID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerID="356b2dba24d8db14d4c0a9d106be62edcfd5b4cc931a374b4c442d3dd1ccaa5e" exitCode=0 Oct 30 23:57:43.843005 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:43.842586 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" event={"ID":"23f642a1-ada6-458a-a8e3-7e5f6014a54f","Type":"ContainerDied","Data":"356b2dba24d8db14d4c0a9d106be62edcfd5b4cc931a374b4c442d3dd1ccaa5e"} Oct 30 23:57:43.965853 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:43.965820 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:44.035681 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.035641 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\") pod \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " Oct 30 23:57:44.035855 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.035690 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\") pod \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " Oct 30 23:57:44.035855 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.035755 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\" (UniqueName: \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\") pod \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\" (UID: \"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd\") " Oct 30 23:57:44.036329 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.036295 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle" pod "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" (UID: "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:44.038112 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.038085 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4" pod "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" (UID: "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd"). InnerVolumeSpecName "kube-api-access-9tbx4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:44.041015 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.040990 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util" pod "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" (UID: "fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:44.137035 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.136956 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9tbx4\" (UniqueName: \"kubernetes.io/projected/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-kube-api-access-9tbx4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:44.137035 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.136987 6234 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:44.137035 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.136996 6234 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd-util\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:44.855519 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.855475 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" event={"ID":"fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd","Type":"ContainerDied","Data":"a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb"} Oct 30 23:57:44.856002 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.855555 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a22c14da89a650352273b796d0e52e1d649cb8b4f2ee02691d40adbaa30c30eb" Oct 30 23:57:44.856002 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.855656 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c7537007944577f64df34960ace3c69136c4162cd56d2c48c7c51e651cxs2gc" Oct 30 23:57:44.977595 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:44.977573 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:45.044683 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.044648 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\" (UniqueName: \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\") pod \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " Oct 30 23:57:45.044866 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.044703 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\") pod \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " Oct 30 23:57:45.044866 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.044738 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\") pod \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\" (UID: \"23f642a1-ada6-458a-a8e3-7e5f6014a54f\") " Oct 30 23:57:45.045157 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.045126 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle" pod "23f642a1-ada6-458a-a8e3-7e5f6014a54f" (UID: "23f642a1-ada6-458a-a8e3-7e5f6014a54f"). InnerVolumeSpecName "bundle". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:45.047086 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.047057 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq" pod "23f642a1-ada6-458a-a8e3-7e5f6014a54f" (UID: "23f642a1-ada6-458a-a8e3-7e5f6014a54f"). InnerVolumeSpecName "kube-api-access-c6fsq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:57:45.048675 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.048652 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util" pod "23f642a1-ada6-458a-a8e3-7e5f6014a54f" (UID: "23f642a1-ada6-458a-a8e3-7e5f6014a54f"). InnerVolumeSpecName "util". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:57:45.145394 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.145311 6234 reconciler_common.go:299] "Volume detached for volume \"bundle\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-bundle\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.145394 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.145341 6234 reconciler_common.go:299] "Volume detached for volume \"util\" (UniqueName: \"kubernetes.io/empty-dir/23f642a1-ada6-458a-a8e3-7e5f6014a54f-util\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.145394 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.145352 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c6fsq\" (UniqueName: \"kubernetes.io/projected/23f642a1-ada6-458a-a8e3-7e5f6014a54f-kube-api-access-c6fsq\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:57:45.861191 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.861164 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" Oct 30 23:57:45.861640 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.861167 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-marketplace/c4959aeb960c8e20c88ac2de924a35b8221494bf772d0dcdd61893e644xbc6c" event={"ID":"23f642a1-ada6-458a-a8e3-7e5f6014a54f","Type":"ContainerDied","Data":"2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2"} Oct 30 23:57:45.861640 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:57:45.861274 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2c17c38489dd8b84d85e1b50c3a15534e379245fced2fde3cc4b4f4097dcaad2" Oct 30 23:58:00.709932 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.709851 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p"] Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710265 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="extract" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710281 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="extract" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710296 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="util" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710304 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="util" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710318 6234 cpu_manager.go:414] "containerMap: removing container" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="pull" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710325 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="pull" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710339 6234 cpu_manager.go:414] "containerMap: removing container" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="extract" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710346 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="extract" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710372 6234 cpu_manager.go:414] "containerMap: removing container" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="util" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710381 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="util" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710389 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="pull" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710397 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="pull" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710463 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="fbc24195-a25b-4ac9-aa24-7e1c10b0e3cd" containerName="extract" Oct 30 23:58:00.710489 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.710475 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="23f642a1-ada6-458a-a8e3-7e5f6014a54f" containerName="extract" Oct 30 23:58:00.714507 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.714488 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.777255 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.777221 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\"" Oct 30 23:58:00.777471 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.777378 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" Oct 30 23:58:00.777548 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.777489 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-475jf\"" Oct 30 23:58:00.777548 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.777492 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" Oct 30 23:58:00.785979 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.785956 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" Oct 30 23:58:00.786099 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.786061 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" Oct 30 23:58:00.822262 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.822202 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p"] Oct 30 23:58:00.873934 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.873897 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.874095 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.873946 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.874095 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.873995 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.874171 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.874095 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkzls\" (UniqueName: \"kubernetes.io/projected/ea19c701-3688-49e2-8623-01fd1391aa28-kube-api-access-gkzls\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.874171 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.874117 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.874250 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.874228 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975584 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975494 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975584 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975546 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975584 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975573 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975895 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975605 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975895 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975634 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkzls\" (UniqueName: \"kubernetes.io/projected/ea19c701-3688-49e2-8623-01fd1391aa28-kube-api-access-gkzls\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.975895 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975663 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.976122 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.975928 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-manifests\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.978171 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.978144 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-metrics-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.978389 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.978370 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ea19c701-3688-49e2-8623-01fd1391aa28-tmp\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.978472 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.978454 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-webhook-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.978583 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.978563 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ea19c701-3688-49e2-8623-01fd1391aa28-apiservice-cert\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:00.989329 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:00.989306 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkzls\" (UniqueName: \"kubernetes.io/projected/ea19c701-3688-49e2-8623-01fd1391aa28-kube-api-access-gkzls\") pod \"opendatahub-operator-controller-manager-58f644b8bd-fc89p\" (UID: \"ea19c701-3688-49e2-8623-01fd1391aa28\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:01.028124 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:01.028095 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:01.357767 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:01.357721 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p"] Oct 30 23:58:01.360537 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:58:01.360509 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea19c701_3688_49e2_8623_01fd1391aa28.slice/crio-ce42e1e0606f4154b8452de3a3b397a3667bcaf3ea1f844c024bcf68d7853b8d WatchSource:0}: Error finding container ce42e1e0606f4154b8452de3a3b397a3667bcaf3ea1f844c024bcf68d7853b8d: Status 404 returned error can't find the container with id ce42e1e0606f4154b8452de3a3b397a3667bcaf3ea1f844c024bcf68d7853b8d Oct 30 23:58:01.921358 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:01.921318 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" event={"ID":"ea19c701-3688-49e2-8623-01fd1391aa28","Type":"ContainerStarted","Data":"ce42e1e0606f4154b8452de3a3b397a3667bcaf3ea1f844c024bcf68d7853b8d"} Oct 30 23:58:03.161534 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:58:03.161496 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:58:05.937604 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:05.937569 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" event={"ID":"ea19c701-3688-49e2-8623-01fd1391aa28","Type":"ContainerStarted","Data":"56c925b1438f29e75e95e2259b491f0513bb0a52eb39a0d2133f65dd277ef5cf"} Oct 30 23:58:06.942921 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:06.942885 6234 generic.go:356] "Generic (PLEG): container finished" podID="ea19c701-3688-49e2-8623-01fd1391aa28" containerID="56c925b1438f29e75e95e2259b491f0513bb0a52eb39a0d2133f65dd277ef5cf" exitCode=0 Oct 30 23:58:06.943364 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:06.942970 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" event={"ID":"ea19c701-3688-49e2-8623-01fd1391aa28","Type":"ContainerDied","Data":"56c925b1438f29e75e95e2259b491f0513bb0a52eb39a0d2133f65dd277ef5cf"} Oct 30 23:58:07.948246 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:07.948209 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" event={"ID":"ea19c701-3688-49e2-8623-01fd1391aa28","Type":"ContainerStarted","Data":"899f69b280b55f2e933531b32a6ea129f108cb4f6e6b3786cf9277768a3a6080"} Oct 30 23:58:07.948629 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:07.948334 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:07.996860 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:07.996814 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" podStartSLOduration=2.9377168620000003 podStartE2EDuration="7.996800253s" totalImagesPullingTime="4.569254808s" totalInitContainerRuntime="489.828583ms" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:07.995982749 +0000 UTC m=+210.029078823" watchObservedRunningTime="2026-10-30 23:58:07.996800253 +0000 UTC m=+210.029896314" Oct 30 23:58:18.953975 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:18.953942 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-58f644b8bd-fc89p" Oct 30 23:58:32.373089 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.373046 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr"] Oct 30 23:58:32.376579 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.376562 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.387509 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.387486 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr"] Oct 30 23:58:32.388728 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.388698 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" Oct 30 23:58:32.388876 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.388745 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" Oct 30 23:58:32.391498 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.391476 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-jfgvc\"" Oct 30 23:58:32.431527 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.431496 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rtrqd\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-kube-api-access-rtrqd\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.431744 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.431538 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.532364 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.532323 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rtrqd\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-kube-api-access-rtrqd\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.532364 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.532366 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.548710 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.548679 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.548710 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.548720 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rtrqd\" (UniqueName: \"kubernetes.io/projected/8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2-kube-api-access-rtrqd\") pod \"cert-manager-cainjector-5c57bbd4fc-hljpr\" (UID: \"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.699253 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.699219 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" Oct 30 23:58:32.810872 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.810820 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-m2xt8"] Oct 30 23:58:32.814141 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.814125 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:32.850975 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.850942 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-m2xt8"] Oct 30 23:58:32.901002 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.900928 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\"" Oct 30 23:58:32.901809 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.901756 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-wgqtv\"" Oct 30 23:58:32.901954 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.901756 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" Oct 30 23:58:32.935394 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.935359 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2g7dx\" (UniqueName: \"kubernetes.io/projected/6521afb3-77b6-4c63-a867-e15f132b1acf-kube-api-access-2g7dx\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:32.935534 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.935428 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6521afb3-77b6-4c63-a867-e15f132b1acf-tls-certs\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:32.935574 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.935544 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6521afb3-77b6-4c63-a867-e15f132b1acf-tmp\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:32.992248 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:32.992167 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr"] Oct 30 23:58:32.995573 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:58:32.995532 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8ab1e6dc_2af8_49cb_b5a9_b367af5e1ff2.slice/crio-5a1280ff6d7e9a007c01473bf41d9c4c076068cff3b7074e42cf50ac8b9ac47a WatchSource:0}: Error finding container 5a1280ff6d7e9a007c01473bf41d9c4c076068cff3b7074e42cf50ac8b9ac47a: Status 404 returned error can't find the container with id 5a1280ff6d7e9a007c01473bf41d9c4c076068cff3b7074e42cf50ac8b9ac47a Oct 30 23:58:33.036077 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.036050 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6521afb3-77b6-4c63-a867-e15f132b1acf-tmp\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.036233 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.036087 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2g7dx\" (UniqueName: \"kubernetes.io/projected/6521afb3-77b6-4c63-a867-e15f132b1acf-kube-api-access-2g7dx\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.036283 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.036233 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6521afb3-77b6-4c63-a867-e15f132b1acf-tls-certs\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.038541 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.038507 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" event={"ID":"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2","Type":"ContainerStarted","Data":"5a1280ff6d7e9a007c01473bf41d9c4c076068cff3b7074e42cf50ac8b9ac47a"} Oct 30 23:58:33.038891 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.038864 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6521afb3-77b6-4c63-a867-e15f132b1acf-tmp\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.039120 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.039104 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6521afb3-77b6-4c63-a867-e15f132b1acf-tls-certs\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.055511 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.055488 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2g7dx\" (UniqueName: \"kubernetes.io/projected/6521afb3-77b6-4c63-a867-e15f132b1acf-kube-api-access-2g7dx\") pod \"kube-auth-proxy-b8699df9-m2xt8\" (UID: \"6521afb3-77b6-4c63-a867-e15f132b1acf\") " pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.125171 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.125134 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" Oct 30 23:58:33.170519 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:58:33.170476 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:58:33.413227 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:33.413198 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-b8699df9-m2xt8"] Oct 30 23:58:33.415433 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:58:33.415403 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6521afb3_77b6_4c63_a867_e15f132b1acf.slice/crio-99c7d50e20e2d98dce770d8da86c53df4dad3e45f5aa6066c94c3ba59579ad4d WatchSource:0}: Error finding container 99c7d50e20e2d98dce770d8da86c53df4dad3e45f5aa6066c94c3ba59579ad4d: Status 404 returned error can't find the container with id 99c7d50e20e2d98dce770d8da86c53df4dad3e45f5aa6066c94c3ba59579ad4d Oct 30 23:58:34.051289 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:34.051194 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" event={"ID":"6521afb3-77b6-4c63-a867-e15f132b1acf","Type":"ContainerStarted","Data":"99c7d50e20e2d98dce770d8da86c53df4dad3e45f5aa6066c94c3ba59579ad4d"} Oct 30 23:58:37.069063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:37.069024 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" event={"ID":"6521afb3-77b6-4c63-a867-e15f132b1acf","Type":"ContainerStarted","Data":"621eb18b150c075312c3f99c9dd090cde24c6a17d9966e4d791372edbc2930da"} Oct 30 23:58:37.091125 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:37.091059 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-b8699df9-m2xt8" podStartSLOduration=2.365612883 podStartE2EDuration="5.091039493s" totalImagesPullingTime="2.72542661s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:37.090129111 +0000 UTC m=+239.123225184" watchObservedRunningTime="2026-10-30 23:58:37.091039493 +0000 UTC m=+239.124135564" Oct 30 23:58:38.077998 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:38.077962 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" event={"ID":"8ab1e6dc-2af8-49cb-b5a9-b367af5e1ff2","Type":"ContainerStarted","Data":"487fcff9e09a39a26412ec6e542eb91202260133c729148e9c6863b2ff3ceefb"} Oct 30 23:58:38.099246 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:58:38.099203 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-hljpr" podStartSLOduration=2.058731618 podStartE2EDuration="6.099189728s" totalImagesPullingTime="4.04045811s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:58:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:58:38.098462429 +0000 UTC m=+240.131558499" watchObservedRunningTime="2026-10-30 23:58:38.099189728 +0000 UTC m=+240.132285789" Oct 30 23:59:03.160549 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:59:03.160509 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:59:29.064290 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.064205 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jr6nm"] Oct 30 23:59:29.067687 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.067667 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.077069 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.077047 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-controller-manager-dockercfg-qd7vs\"" Oct 30 23:59:29.081936 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.081915 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\"" Oct 30 23:59:29.083821 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.083775 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\"" Oct 30 23:59:29.083972 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.083956 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"kserve-webhook-server-cert\"" Oct 30 23:59:29.084968 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.084946 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jr6nm"] Oct 30 23:59:29.094751 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.094727 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kbd99\" (UniqueName: \"kubernetes.io/projected/130cd323-4efe-41e8-878a-d3b25ea31917-kube-api-access-kbd99\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.094885 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.094774 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130cd323-4efe-41e8-878a-d3b25ea31917-cert\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.121192 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.121166 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxsw6"] Oct 30 23:59:29.124415 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.124397 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.142510 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.142487 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxsw6"] Oct 30 23:59:29.149091 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.149068 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/odh-model-controller-769875db56-mpwvn"] Oct 30 23:59:29.152205 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.152189 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.166484 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.166459 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mpwvn"] Oct 30 23:59:29.177393 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.177374 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\"" Oct 30 23:59:29.177726 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.177700 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-webhook-cert\"" Oct 30 23:59:29.177865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.177738 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-khglh\"" Oct 30 23:59:29.177865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.177775 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-metrics-tls\"" Oct 30 23:59:29.177865 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.177775 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"odh-model-controller-dockercfg-x597m\"" Oct 30 23:59:29.195426 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195395 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-cert\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.195426 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195429 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6jvpl\" (UniqueName: \"kubernetes.io/projected/6124af8d-fa59-479c-b6c0-561a759e7e71-kube-api-access-6jvpl\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.195625 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195448 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-metrics-certs\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.195625 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195466 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvnwm\" (UniqueName: \"kubernetes.io/projected/b9dad48a-5c54-4db3-a98e-581219b1046c-kube-api-access-bvnwm\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.195625 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195534 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9dad48a-5c54-4db3-a98e-581219b1046c-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.195757 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195621 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kbd99\" (UniqueName: \"kubernetes.io/projected/130cd323-4efe-41e8-878a-d3b25ea31917-kube-api-access-kbd99\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.195757 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.195654 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130cd323-4efe-41e8-878a-d3b25ea31917-cert\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.198394 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.198372 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/130cd323-4efe-41e8-878a-d3b25ea31917-cert\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.209578 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.209551 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kbd99\" (UniqueName: \"kubernetes.io/projected/130cd323-4efe-41e8-878a-d3b25ea31917-kube-api-access-kbd99\") pod \"kserve-controller-manager-776cc66d47-jr6nm\" (UID: \"130cd323-4efe-41e8-878a-d3b25ea31917\") " pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.296633 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.296603 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-cert\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.296633 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.296639 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6jvpl\" (UniqueName: \"kubernetes.io/projected/6124af8d-fa59-479c-b6c0-561a759e7e71-kube-api-access-6jvpl\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.296909 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.296658 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-metrics-certs\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.296909 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.296673 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvnwm\" (UniqueName: \"kubernetes.io/projected/b9dad48a-5c54-4db3-a98e-581219b1046c-kube-api-access-bvnwm\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.296909 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.296695 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9dad48a-5c54-4db3-a98e-581219b1046c-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.299328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.299295 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b9dad48a-5c54-4db3-a98e-581219b1046c-tls-certs\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.299328 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.299316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-cert\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.299540 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.299368 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6124af8d-fa59-479c-b6c0-561a759e7e71-metrics-certs\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.310071 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.310048 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvnwm\" (UniqueName: \"kubernetes.io/projected/b9dad48a-5c54-4db3-a98e-581219b1046c-kube-api-access-bvnwm\") pod \"model-serving-api-75bd89fdfc-sxsw6\" (UID: \"b9dad48a-5c54-4db3-a98e-581219b1046c\") " pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.310257 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.310236 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6jvpl\" (UniqueName: \"kubernetes.io/projected/6124af8d-fa59-479c-b6c0-561a759e7e71-kube-api-access-6jvpl\") pod \"odh-model-controller-769875db56-mpwvn\" (UID: \"6124af8d-fa59-479c-b6c0-561a759e7e71\") " pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.373220 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.373142 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc"] Oct 30 23:59:29.377510 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.377490 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.380397 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.380376 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:29.390258 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.390234 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc"] Oct 30 23:59:29.398019 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.397990 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398134 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398037 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de47760d-2eff-44ce-8aa5-9d82b9842a65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398175 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398142 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398220 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398194 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398278 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398260 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5ljx5\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-kube-api-access-5ljx5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398364 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398349 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.398405 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.398392 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.436333 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.436084 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:29.464045 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.463999 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:29.501274 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501242 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501396 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501293 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de47760d-2eff-44ce-8aa5-9d82b9842a65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501396 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501333 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501396 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501363 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501501 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501429 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5ljx5\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-kube-api-access-5ljx5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501501 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501473 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.501563 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.501507 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.502577 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.502247 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.505422 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.505373 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.505629 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.505610 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.505697 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.505679 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.505927 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.505907 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/de47760d-2eff-44ce-8aa5-9d82b9842a65-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.515584 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.515552 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.518476 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.518452 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5ljx5\" (UniqueName: \"kubernetes.io/projected/de47760d-2eff-44ce-8aa5-9d82b9842a65-kube-api-access-5ljx5\") pod \"istiod-openshift-gateway-7ddd9cfdc8-qzxwc\" (UID: \"de47760d-2eff-44ce-8aa5-9d82b9842a65\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.657684 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.657657 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/kserve-controller-manager-776cc66d47-jr6nm"] Oct 30 23:59:29.660337 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:59:29.660308 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod130cd323_4efe_41e8_878a_d3b25ea31917.slice/crio-cf315cac17b87705d20453dbb3af6332d08b5942865afa787c7f8be7b05da16f WatchSource:0}: Error finding container cf315cac17b87705d20453dbb3af6332d08b5942865afa787c7f8be7b05da16f: Status 404 returned error can't find the container with id cf315cac17b87705d20453dbb3af6332d08b5942865afa787c7f8be7b05da16f Oct 30 23:59:29.689870 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.689848 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:29.715545 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.715514 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-75bd89fdfc-sxsw6"] Oct 30 23:59:29.719085 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:59:29.719061 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb9dad48a_5c54_4db3_a98e_581219b1046c.slice/crio-4873782ca670440f05733f4ae03ec98d7fdcb3cefeaeadc19541cd7b81a2203a WatchSource:0}: Error finding container 4873782ca670440f05733f4ae03ec98d7fdcb3cefeaeadc19541cd7b81a2203a: Status 404 returned error can't find the container with id 4873782ca670440f05733f4ae03ec98d7fdcb3cefeaeadc19541cd7b81a2203a Oct 30 23:59:29.751136 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.749768 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/odh-model-controller-769875db56-mpwvn"] Oct 30 23:59:29.752237 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:59:29.752206 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6124af8d_fa59_479c_b6c0_561a759e7e71.slice/crio-fe8b72d1d0c00e7de5ce67711b2a550be55e4858d351ec8724f142cb17b076b5 WatchSource:0}: Error finding container fe8b72d1d0c00e7de5ce67711b2a550be55e4858d351ec8724f142cb17b076b5: Status 404 returned error can't find the container with id fe8b72d1d0c00e7de5ce67711b2a550be55e4858d351ec8724f142cb17b076b5 Oct 30 23:59:29.969048 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.969011 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc"] Oct 30 23:59:29.971899 ip-10-0-130-152 kubenswrapper[6234]: W1030 23:59:29.971866 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podde47760d_2eff_44ce_8aa5_9d82b9842a65.slice/crio-bf8570661fe81d0b99054c80e8e4c07ccfcccf41a3ee748d11f8cf6bc2535c0d WatchSource:0}: Error finding container bf8570661fe81d0b99054c80e8e4c07ccfcccf41a3ee748d11f8cf6bc2535c0d: Status 404 returned error can't find the container with id bf8570661fe81d0b99054c80e8e4c07ccfcccf41a3ee748d11f8cf6bc2535c0d Oct 30 23:59:29.974010 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.973981 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 23:59:29.974081 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:29.974039 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 30 23:59:30.290357 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.290250 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mpwvn" event={"ID":"6124af8d-fa59-479c-b6c0-561a759e7e71","Type":"ContainerStarted","Data":"fe8b72d1d0c00e7de5ce67711b2a550be55e4858d351ec8724f142cb17b076b5"} Oct 30 23:59:30.292241 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.292209 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" event={"ID":"b9dad48a-5c54-4db3-a98e-581219b1046c","Type":"ContainerStarted","Data":"4873782ca670440f05733f4ae03ec98d7fdcb3cefeaeadc19541cd7b81a2203a"} Oct 30 23:59:30.299231 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.299198 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" event={"ID":"de47760d-2eff-44ce-8aa5-9d82b9842a65","Type":"ContainerStarted","Data":"a708cfc10b7071b47ce514b27c94cee4b09baf8646f22a07d324cc57d0d31d85"} Oct 30 23:59:30.299386 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.299255 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" event={"ID":"de47760d-2eff-44ce-8aa5-9d82b9842a65","Type":"ContainerStarted","Data":"bf8570661fe81d0b99054c80e8e4c07ccfcccf41a3ee748d11f8cf6bc2535c0d"} Oct 30 23:59:30.299589 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.299566 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:30.302585 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.302458 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" event={"ID":"130cd323-4efe-41e8-878a-d3b25ea31917","Type":"ContainerStarted","Data":"cf315cac17b87705d20453dbb3af6332d08b5942865afa787c7f8be7b05da16f"} Oct 30 23:59:30.331438 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:30.331374 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" podStartSLOduration=1.331355149 podStartE2EDuration="1.331355149s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:30.327696843 +0000 UTC m=+292.360792915" watchObservedRunningTime="2026-10-30 23:59:30.331355149 +0000 UTC m=+292.364451220" Oct 30 23:59:31.310817 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.310711 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-qzxwc" Oct 30 23:59:31.376222 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.376184 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:59:31.376446 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.376420 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" podUID="d747c179-cb07-4b21-be46-208342311286" containerName="discovery" containerID="cri-o://c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37" gracePeriod=30 Oct 30 23:59:31.712286 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.712025 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:59:31.729519 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729487 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729691 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729554 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729691 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729603 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729691 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729652 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729887 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729698 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729887 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729738 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.729887 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.729804 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\") pod \"d747c179-cb07-4b21-be46-208342311286\" (UID: \"d747c179-cb07-4b21-be46-208342311286\") " Oct 30 23:59:31.731494 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.731463 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 30 23:59:31.733268 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.733242 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:31.734875 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.734848 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "kube-api-access-4sxh6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:31.735097 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.735073 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:31.738118 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.738092 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 30 23:59:31.738224 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.738198 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 30 23:59:31.744607 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.744578 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token" pod "d747c179-cb07-4b21-be46-208342311286" (UID: "d747c179-cb07-4b21-be46-208342311286"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 30 23:59:31.831485 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831447 6234 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-cacerts\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831485 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831488 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/d747c179-cb07-4b21-be46-208342311286-istio-csr-ca-configmap\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831734 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831501 6234 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-kubeconfig\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831734 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831511 6234 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/d747c179-cb07-4b21-be46-208342311286-istio-csr-dns-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831734 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831519 6234 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/d747c179-cb07-4b21-be46-208342311286-local-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831734 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831528 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:31.831734 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:31.831541 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4sxh6\" (UniqueName: \"kubernetes.io/projected/d747c179-cb07-4b21-be46-208342311286-kube-api-access-4sxh6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 30 23:59:32.316206 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.316171 6234 generic.go:356] "Generic (PLEG): container finished" podID="d747c179-cb07-4b21-be46-208342311286" containerID="c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37" exitCode=0 Oct 30 23:59:32.316800 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.316762 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" Oct 30 23:59:32.316906 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.316849 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" event={"ID":"d747c179-cb07-4b21-be46-208342311286","Type":"ContainerDied","Data":"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37"} Oct 30 23:59:32.316948 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.316908 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8" event={"ID":"d747c179-cb07-4b21-be46-208342311286","Type":"ContainerDied","Data":"c14976152f8689c379929e9524385bb5616d43206d1de659a2794ef14b4fb3a9"} Oct 30 23:59:32.316948 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.316934 6234 scope.go:122] "RemoveContainer" containerID="c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37" Oct 30 23:59:32.327010 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.326992 6234 scope.go:122] "RemoveContainer" containerID="c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37" Oct 30 23:59:32.327367 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:59:32.327340 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37\": container with ID starting with c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37 not found: ID does not exist" containerID="c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37" Oct 30 23:59:32.327518 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.327382 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37"} err="failed to get container status \"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37\": rpc error: code = NotFound desc = could not find container \"c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37\": container with ID starting with c5813dc4fedf8eebba895b57ca7f3280e45942f638a57536c7fab379ff925a37 not found: ID does not exist" Oct 30 23:59:32.341497 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.341431 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:59:32.346184 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:32.346155 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7876bfcb74-8rgd8"] Oct 30 23:59:33.160026 ip-10-0-130-152 kubenswrapper[6234]: E1030 23:59:33.159990 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 30 23:59:33.325580 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.325544 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/odh-model-controller-769875db56-mpwvn" event={"ID":"6124af8d-fa59-479c-b6c0-561a759e7e71","Type":"ContainerStarted","Data":"8df4a9a7fd43182399abcf27e5c8c9450b49bcbf3a15f882acf9de07049b844f"} Oct 30 23:59:33.326063 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.325623 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:33.327205 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.327166 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" event={"ID":"b9dad48a-5c54-4db3-a98e-581219b1046c","Type":"ContainerStarted","Data":"7aaa02ef1d3761488819e1b94b68d396cb4e758d84686be5a2406a72eaf5b286"} Oct 30 23:59:33.327329 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.327219 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 30 23:59:33.328509 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.328491 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" event={"ID":"130cd323-4efe-41e8-878a-d3b25ea31917","Type":"ContainerStarted","Data":"4c0bfcea9476300ddc0bf5e2d5559f6a597efac0344d12cdfb83bde4a094bacb"} Oct 30 23:59:33.328645 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.328628 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 30 23:59:33.375607 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.375449 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" podStartSLOduration=1.460249139 podStartE2EDuration="4.375432466s" totalImagesPullingTime="2.915183327s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:33.375032973 +0000 UTC m=+295.408129046" watchObservedRunningTime="2026-10-30 23:59:33.375432466 +0000 UTC m=+295.408528548" Oct 30 23:59:33.376224 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.376184 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/odh-model-controller-769875db56-mpwvn" podStartSLOduration=0.949621058 podStartE2EDuration="4.376175007s" totalImagesPullingTime="3.426553949s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:33.351967668 +0000 UTC m=+295.385063739" watchObservedRunningTime="2026-10-30 23:59:33.376175007 +0000 UTC m=+295.409271080" Oct 30 23:59:33.398933 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:33.398885 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" podStartSLOduration=1.797488931 podStartE2EDuration="4.398870751s" totalImagesPullingTime="2.60138182s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-30 23:59:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-30 23:59:33.39774514 +0000 UTC m=+295.430841224" watchObservedRunningTime="2026-10-30 23:59:33.398870751 +0000 UTC m=+295.431966821" Oct 30 23:59:34.079285 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:34.079253 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d747c179-cb07-4b21-be46-208342311286" path="/var/lib/kubelet/pods/d747c179-cb07-4b21-be46-208342311286/volumes" Oct 30 23:59:44.335467 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:44.335438 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/odh-model-controller-769875db56-mpwvn" Oct 30 23:59:44.337562 ip-10-0-130-152 kubenswrapper[6234]: I1030 23:59:44.337541 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-75bd89fdfc-sxsw6" Oct 31 00:00:03.161153 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:00:03.161116 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:00:04.338056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:00:04.338027 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/kserve-controller-manager-776cc66d47-jr6nm" Oct 31 00:00:33.160439 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:00:33.160398 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:01:03.161177 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:01:03.161097 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:01:33.160071 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:01:33.160029 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:02:03.160617 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:02:03.160582 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:02:33.160382 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:02:33.160291 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:03:03.161037 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:03:03.160993 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:03:33.160230 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:03:33.160188 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:04:03.160520 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:04:03.160438 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:04:33.160471 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:04:33.160436 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:05:03.159619 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:05:03.159584 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:05:33.160315 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:05:33.160234 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:06:03.160441 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:06:03.160399 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:06:33.160358 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:06:33.160315 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:07:03.160465 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:07:03.160385 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:07:33.160433 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:07:33.160391 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:08:03.160396 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:08:03.160350 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:08:33.159813 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:08:33.159752 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:09:03.160582 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:09:03.160539 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:09:12.299155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.299116 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:12.299616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.299452 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d747c179-cb07-4b21-be46-208342311286" containerName="discovery" Oct 31 00:09:12.299616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.299463 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d747c179-cb07-4b21-be46-208342311286" containerName="discovery" Oct 31 00:09:12.299616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.299524 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d747c179-cb07-4b21-be46-208342311286" containerName="discovery" Oct 31 00:09:12.302401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.302381 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.310291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.310271 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-jkqzl\"" Oct 31 00:09:12.312068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.312049 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\"" Oct 31 00:09:12.312141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.312115 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" Oct 31 00:09:12.312797 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.312727 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" Oct 31 00:09:12.314292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.314271 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:12.381822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.381746 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.382056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.382033 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jz5q6\" (UniqueName: \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.402608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.402575 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:12.482586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.482555 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.482586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.482587 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jz5q6\" (UniqueName: \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.483180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.483160 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.495931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.495901 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jz5q6\" (UniqueName: \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\") pod \"limitador-limitador-74dfcfb894-wktj2\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.614383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.614292 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:12.888129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.888106 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:12.890941 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:09:12.890916 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5dde707d_2a13_43f3_a7ec_12316cbf33e3.slice/crio-c97cbfc67924c2b1a3584f87f93bc148bbed9262ea2e5655dc76a95c46d51ce5 WatchSource:0}: Error finding container c97cbfc67924c2b1a3584f87f93bc148bbed9262ea2e5655dc76a95c46d51ce5: Status 404 returned error can't find the container with id c97cbfc67924c2b1a3584f87f93bc148bbed9262ea2e5655dc76a95c46d51ce5 Oct 31 00:09:12.893214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:12.893199 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:09:13.468312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.468277 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:09:13.472741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.472718 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:09:13.479596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.479574 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:09:13.480151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.480132 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-gzz8j\"" Oct 31 00:09:13.490277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.490253 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njcpx\" (UniqueName: \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\") pod \"authorino-56c6c9ddd5-67jtl\" (UID: \"09669322-f2c4-43b7-9a8e-5285a64f75a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:09:13.590766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.590732 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njcpx\" (UniqueName: \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\") pod \"authorino-56c6c9ddd5-67jtl\" (UID: \"09669322-f2c4-43b7-9a8e-5285a64f75a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:09:13.604691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.604659 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-njcpx\" (UniqueName: \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\") pod \"authorino-56c6c9ddd5-67jtl\" (UID: \"09669322-f2c4-43b7-9a8e-5285a64f75a1\") " pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:09:13.704964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.704922 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" event={"ID":"5dde707d-2a13-43f3-a7ec-12316cbf33e3","Type":"ContainerStarted","Data":"c97cbfc67924c2b1a3584f87f93bc148bbed9262ea2e5655dc76a95c46d51ce5"} Oct 31 00:09:13.784807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:13.784705 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:09:14.056557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:14.056481 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:09:14.059839 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:09:14.059813 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09669322_f2c4_43b7_9a8e_5285a64f75a1.slice/crio-597612b6390d83432f328d40e8551af99f19442a88a52bd14e41d963a469eb8b WatchSource:0}: Error finding container 597612b6390d83432f328d40e8551af99f19442a88a52bd14e41d963a469eb8b: Status 404 returned error can't find the container with id 597612b6390d83432f328d40e8551af99f19442a88a52bd14e41d963a469eb8b Oct 31 00:09:14.711119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:14.711080 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" event={"ID":"09669322-f2c4-43b7-9a8e-5285a64f75a1","Type":"ContainerStarted","Data":"597612b6390d83432f328d40e8551af99f19442a88a52bd14e41d963a469eb8b"} Oct 31 00:09:16.724123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:16.724076 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" event={"ID":"5dde707d-2a13-43f3-a7ec-12316cbf33e3","Type":"ContainerStarted","Data":"28346830fb038779a01b6075e60fc6d287fbae9af39b139a3935e087427e7a6d"} Oct 31 00:09:16.724652 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:16.724212 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:16.744828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:16.744748 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" podStartSLOduration=1.939609897 podStartE2EDuration="4.744734273s" totalImagesPullingTime="2.805124376s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:16.74367227 +0000 UTC m=+878.776768341" watchObservedRunningTime="2026-10-31 00:09:16.744734273 +0000 UTC m=+878.777830343" Oct 31 00:09:17.730111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:17.730080 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" event={"ID":"09669322-f2c4-43b7-9a8e-5285a64f75a1","Type":"ContainerStarted","Data":"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac"} Oct 31 00:09:17.749037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:17.748991 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" podStartSLOduration=1.993925814 podStartE2EDuration="4.748977448s" totalImagesPullingTime="2.755051634s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:17.748291962 +0000 UTC m=+879.781388031" watchObservedRunningTime="2026-10-31 00:09:17.748977448 +0000 UTC m=+879.782073517" Oct 31 00:09:26.221391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:26.221355 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:26.221901 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:26.221541 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" podUID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" containerName="limitador" containerID="cri-o://28346830fb038779a01b6075e60fc6d287fbae9af39b139a3935e087427e7a6d" gracePeriod=30 Oct 31 00:09:26.222314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:26.222294 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:26.766682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:26.766647 6234 generic.go:356] "Generic (PLEG): container finished" podID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" containerID="28346830fb038779a01b6075e60fc6d287fbae9af39b139a3935e087427e7a6d" exitCode=0 Oct 31 00:09:26.766868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:26.766694 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" event={"ID":"5dde707d-2a13-43f3-a7ec-12316cbf33e3","Type":"ContainerDied","Data":"28346830fb038779a01b6075e60fc6d287fbae9af39b139a3935e087427e7a6d"} Oct 31 00:09:27.162903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.162875 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:27.201738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.199459 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\" (UniqueName: \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\") pod \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " Oct 31 00:09:27.201738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.199564 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\" (UniqueName: \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\") pod \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\" (UID: \"5dde707d-2a13-43f3-a7ec-12316cbf33e3\") " Oct 31 00:09:27.201738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.200217 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file" pod "5dde707d-2a13-43f3-a7ec-12316cbf33e3" (UID: "5dde707d-2a13-43f3-a7ec-12316cbf33e3"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:09:27.206014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.205978 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6" pod "5dde707d-2a13-43f3-a7ec-12316cbf33e3" (UID: "5dde707d-2a13-43f3-a7ec-12316cbf33e3"). InnerVolumeSpecName "kube-api-access-jz5q6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:09:27.300662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.300585 6234 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/5dde707d-2a13-43f3-a7ec-12316cbf33e3-config-file\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:27.300662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.300613 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jz5q6\" (UniqueName: \"kubernetes.io/projected/5dde707d-2a13-43f3-a7ec-12316cbf33e3-kube-api-access-jz5q6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:09:27.772457 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.772422 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" event={"ID":"5dde707d-2a13-43f3-a7ec-12316cbf33e3","Type":"ContainerDied","Data":"c97cbfc67924c2b1a3584f87f93bc148bbed9262ea2e5655dc76a95c46d51ce5"} Oct 31 00:09:27.772653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.772506 6234 scope.go:122] "RemoveContainer" containerID="28346830fb038779a01b6075e60fc6d287fbae9af39b139a3935e087427e7a6d" Oct 31 00:09:27.772653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.772433 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-wktj2" Oct 31 00:09:27.795731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.795705 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:27.799863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:27.799842 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-wktj2"] Oct 31 00:09:28.079592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.079517 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" path="/var/lib/kubelet/pods/5dde707d-2a13-43f3-a7ec-12316cbf33e3/volumes" Oct 31 00:09:28.730641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.730609 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz"] Oct 31 00:09:28.731043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.730978 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" containerName="limitador" Oct 31 00:09:28.731043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.730989 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" containerName="limitador" Oct 31 00:09:28.731121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.731066 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5dde707d-2a13-43f3-a7ec-12316cbf33e3" containerName="limitador" Oct 31 00:09:28.735268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.735249 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.742241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.742185 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz"] Oct 31 00:09:28.743773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.743748 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" Oct 31 00:09:28.744012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.743991 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"postgres-creds\"" Oct 31 00:09:28.744012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.744002 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" Oct 31 00:09:28.744253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.744236 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"default-dockercfg-gjcs5\"" Oct 31 00:09:28.812079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.812050 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a721706-8e9c-454a-81d0-19e4d564eebf-data\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.812255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.812126 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbkg2\" (UniqueName: \"kubernetes.io/projected/4a721706-8e9c-454a-81d0-19e4d564eebf-kube-api-access-qbkg2\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.913529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.913492 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbkg2\" (UniqueName: \"kubernetes.io/projected/4a721706-8e9c-454a-81d0-19e4d564eebf-kube-api-access-qbkg2\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.913529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.913533 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a721706-8e9c-454a-81d0-19e4d564eebf-data\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.914002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.913984 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data\" (UniqueName: \"kubernetes.io/empty-dir/4a721706-8e9c-454a-81d0-19e4d564eebf-data\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:28.926684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:28.926653 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbkg2\" (UniqueName: \"kubernetes.io/projected/4a721706-8e9c-454a-81d0-19e4d564eebf-kube-api-access-qbkg2\") pod \"postgres-7cdbf6dc44-ngrsz\" (UID: \"4a721706-8e9c-454a-81d0-19e4d564eebf\") " pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:29.049833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:29.049714 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:29.328587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:29.328468 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz"] Oct 31 00:09:29.331096 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:09:29.331071 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4a721706_8e9c_454a_81d0_19e4d564eebf.slice/crio-bcf3f493044d75cdba879633b21a321512ccfaae05e407ca3451139380a6db07 WatchSource:0}: Error finding container bcf3f493044d75cdba879633b21a321512ccfaae05e407ca3451139380a6db07: Status 404 returned error can't find the container with id bcf3f493044d75cdba879633b21a321512ccfaae05e407ca3451139380a6db07 Oct 31 00:09:29.783235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:29.783200 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" event={"ID":"4a721706-8e9c-454a-81d0-19e4d564eebf","Type":"ContainerStarted","Data":"bcf3f493044d75cdba879633b21a321512ccfaae05e407ca3451139380a6db07"} Oct 31 00:09:33.162040 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:09:33.161991 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:09:35.814531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:35.814497 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" event={"ID":"4a721706-8e9c-454a-81d0-19e4d564eebf","Type":"ContainerStarted","Data":"5e46c785691926a4d20f8bb22cc78e87eede8fb0b167032ebbf64c335d8f691b"} Oct 31 00:09:35.815004 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:35.814571 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:35.834596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:35.834549 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" podStartSLOduration=1.837032911 podStartE2EDuration="7.834534677s" totalImagesPullingTime="5.997501766s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:28 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:09:35.833365384 +0000 UTC m=+897.866461454" watchObservedRunningTime="2026-10-31 00:09:35.834534677 +0000 UTC m=+897.867630746" Oct 31 00:09:41.847867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:41.847836 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/postgres-7cdbf6dc44-ngrsz" Oct 31 00:09:54.225833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.225716 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk"] Oct 31 00:09:54.233068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.233040 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.242536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.241543 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk"] Oct 31 00:09:54.243087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.243066 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-tls\"" Oct 31 00:09:54.244611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.244542 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-dockercfg-npg6v\"" Oct 31 00:09:54.244611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.244581 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-logs-tenancy-proxy-ca-bundle\"" Oct 31 00:09:54.244611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.244588 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"usage-tenancy-proxy-source\"" Oct 31 00:09:54.360953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.360917 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t5z5w\" (UniqueName: \"kubernetes.io/projected/5b700d0c-b689-4767-9b25-aa14f4131036-kube-api-access-t5z5w\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.360953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.360958 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.361169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.360981 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b700d0c-b689-4767-9b25-aa14f4131036-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.361169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.361084 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.461591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.461553 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.461766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.461630 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t5z5w\" (UniqueName: \"kubernetes.io/projected/5b700d0c-b689-4767-9b25-aa14f4131036-kube-api-access-t5z5w\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.461766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.461659 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.461766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.461684 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b700d0c-b689-4767-9b25-aa14f4131036-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.462374 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.462315 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-service-ca\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.462603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.462538 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proxy-source\" (UniqueName: \"kubernetes.io/configmap/5b700d0c-b689-4767-9b25-aa14f4131036-proxy-source\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.464318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.464295 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/5b700d0c-b689-4767-9b25-aa14f4131036-tls-cert\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.475217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.475193 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t5z5w\" (UniqueName: \"kubernetes.io/projected/5b700d0c-b689-4767-9b25-aa14f4131036-kube-api-access-t5z5w\") pod \"usage-logs-tenancy-proxy-9c5d49bcd-4w2nk\" (UID: \"5b700d0c-b689-4767-9b25-aa14f4131036\") " pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.550693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.550613 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:09:54.826608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.826579 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk"] Oct 31 00:09:54.828128 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:09:54.828099 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b700d0c_b689_4767_9b25_aa14f4131036.slice/crio-02038ae5e9554e20ddddd53df55e966acee54eacc8b7fe05a17dbec2a0c53111 WatchSource:0}: Error finding container 02038ae5e9554e20ddddd53df55e966acee54eacc8b7fe05a17dbec2a0c53111: Status 404 returned error can't find the container with id 02038ae5e9554e20ddddd53df55e966acee54eacc8b7fe05a17dbec2a0c53111 Oct 31 00:09:54.893770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:09:54.893738 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" event={"ID":"5b700d0c-b689-4767-9b25-aa14f4131036","Type":"ContainerStarted","Data":"02038ae5e9554e20ddddd53df55e966acee54eacc8b7fe05a17dbec2a0c53111"} Oct 31 00:10:03.163270 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:10:03.163216 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:10:14.984230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:14.984195 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" event={"ID":"5b700d0c-b689-4767-9b25-aa14f4131036","Type":"ContainerStarted","Data":"1c3946a43d71798e22d57f17870940a04fb9058831d06dad3221308f23f39e0d"} Oct 31 00:10:14.984621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:14.984298 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:10:15.005631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:15.005580 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" podStartSLOduration=1.87262504 podStartE2EDuration="21.005565352s" totalImagesPullingTime="19.132940312s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:09:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:15.004565738 +0000 UTC m=+937.037661810" watchObservedRunningTime="2026-10-31 00:10:15.005565352 +0000 UTC m=+937.038661470" Oct 31 00:10:25.344373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.344338 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:25.347555 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.347533 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.358114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.358093 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" Oct 31 00:10:25.358114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.358093 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-ldwtt\"" Oct 31 00:10:25.358285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.358211 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\"" Oct 31 00:10:25.361290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.361271 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:25.441734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.441696 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.441734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.441737 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4hqvd\" (UniqueName: \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.441994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.441759 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.441994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.441877 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.543361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.543325 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.543361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.543365 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4hqvd\" (UniqueName: \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.543617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.543386 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.543617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.543408 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.544036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.544013 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.546070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.546043 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.546161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.546101 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.561444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.561416 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4hqvd\" (UniqueName: \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\") pod \"maas-api-6d669c7fcd-jvrjr\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.660632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.660558 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:25.939525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.939496 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:25.942326 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:10:25.942300 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod381200eb_7e51_49ba_a281_2c8117714e85.slice/crio-fe2fb7beae55522ccaae206ea1ecc466da6facbb01c44e25f08c6c681eb38398 WatchSource:0}: Error finding container fe2fb7beae55522ccaae206ea1ecc466da6facbb01c44e25f08c6c681eb38398: Status 404 returned error can't find the container with id fe2fb7beae55522ccaae206ea1ecc466da6facbb01c44e25f08c6c681eb38398 Oct 31 00:10:25.992213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:25.992184 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/usage-logs-tenancy-proxy-9c5d49bcd-4w2nk" Oct 31 00:10:26.031641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:26.031599 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" event={"ID":"381200eb-7e51-49ba-a281-2c8117714e85","Type":"ContainerStarted","Data":"fe2fb7beae55522ccaae206ea1ecc466da6facbb01c44e25f08c6c681eb38398"} Oct 31 00:10:29.051262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:29.051222 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" event={"ID":"381200eb-7e51-49ba-a281-2c8117714e85","Type":"ContainerStarted","Data":"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc"} Oct 31 00:10:29.051752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:29.051294 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:29.073769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:29.073723 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" podStartSLOduration=1.39543613 podStartE2EDuration="4.073711248s" totalImagesPullingTime="2.678275118s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:29.073416409 +0000 UTC m=+951.106512482" watchObservedRunningTime="2026-10-31 00:10:29.073711248 +0000 UTC m=+951.106807317" Oct 31 00:10:33.159596 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:10:33.159559 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:10:35.061608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:35.061578 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:36.020686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.020648 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:10:36.267419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.267388 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.273662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.273593 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:10:36.447098 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.447058 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk7cf\" (UniqueName: \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.447260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.447126 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.447260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.447234 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.447335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.447264 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.548018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.547934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.548018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.547999 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.548018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.548023 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.548233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.548061 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk7cf\" (UniqueName: \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.548751 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.548682 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.550609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.550586 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.550703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.550681 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.562383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.562356 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk7cf\" (UniqueName: \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\") pod \"maas-api-6676c7f7bc-7f6dp\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.594360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.594323 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:36.916947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:36.916743 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:10:36.918986 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:10:36.918954 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20fc4c02_e14c_42c4_bad5_8858afedc5d5.slice/crio-d800ac8a624c1d9ea0a47e38f6cac96e200f811eaa0530293538a5a4b8415780 WatchSource:0}: Error finding container d800ac8a624c1d9ea0a47e38f6cac96e200f811eaa0530293538a5a4b8415780: Status 404 returned error can't find the container with id d800ac8a624c1d9ea0a47e38f6cac96e200f811eaa0530293538a5a4b8415780 Oct 31 00:10:37.087220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:37.087186 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" event={"ID":"20fc4c02-e14c-42c4-bad5-8858afedc5d5","Type":"ContainerStarted","Data":"d800ac8a624c1d9ea0a47e38f6cac96e200f811eaa0530293538a5a4b8415780"} Oct 31 00:10:38.091852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:38.091820 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" event={"ID":"20fc4c02-e14c-42c4-bad5-8858afedc5d5","Type":"ContainerStarted","Data":"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af"} Oct 31 00:10:38.092249 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:38.091928 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:38.115221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:38.115172 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" podStartSLOduration=1.575066596 podStartE2EDuration="2.11515993s" totalImagesPullingTime="540.093334ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:10:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:10:38.113878979 +0000 UTC m=+960.146975053" watchObservedRunningTime="2026-10-31 00:10:38.11515993 +0000 UTC m=+960.148256000" Oct 31 00:10:38.742013 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:38.741974 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:10:38.742213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:38.742172 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" podUID="09669322-f2c4-43b7-9a8e-5285a64f75a1" containerName="authorino" containerID="cri-o://903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac" gracePeriod=30 Oct 31 00:10:39.033992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.033961 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:10:39.097505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.097474 6234 generic.go:356] "Generic (PLEG): container finished" podID="09669322-f2c4-43b7-9a8e-5285a64f75a1" containerID="903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac" exitCode=0 Oct 31 00:10:39.097967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.097539 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" Oct 31 00:10:39.097967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.097562 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" event={"ID":"09669322-f2c4-43b7-9a8e-5285a64f75a1","Type":"ContainerDied","Data":"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac"} Oct 31 00:10:39.097967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.097602 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-56c6c9ddd5-67jtl" event={"ID":"09669322-f2c4-43b7-9a8e-5285a64f75a1","Type":"ContainerDied","Data":"597612b6390d83432f328d40e8551af99f19442a88a52bd14e41d963a469eb8b"} Oct 31 00:10:39.097967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.097623 6234 scope.go:122] "RemoveContainer" containerID="903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac" Oct 31 00:10:39.106085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.106068 6234 scope.go:122] "RemoveContainer" containerID="903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac" Oct 31 00:10:39.106387 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:10:39.106366 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac\": container with ID starting with 903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac not found: ID does not exist" containerID="903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac" Oct 31 00:10:39.106435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.106395 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac"} err="failed to get container status \"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac\": rpc error: code = NotFound desc = could not find container \"903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac\": container with ID starting with 903d4a1c6d933d383dfdb6c51f8d948ba02ab92d3cca7fd6412bc49a3507d1ac not found: ID does not exist" Oct 31 00:10:39.170366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.170329 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\" (UniqueName: \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\") pod \"09669322-f2c4-43b7-9a8e-5285a64f75a1\" (UID: \"09669322-f2c4-43b7-9a8e-5285a64f75a1\") " Oct 31 00:10:39.172485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.172456 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx" pod "09669322-f2c4-43b7-9a8e-5285a64f75a1" (UID: "09669322-f2c4-43b7-9a8e-5285a64f75a1"). InnerVolumeSpecName "kube-api-access-njcpx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:39.271610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.271524 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njcpx\" (UniqueName: \"kubernetes.io/projected/09669322-f2c4-43b7-9a8e-5285a64f75a1-kube-api-access-njcpx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:39.420130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.420098 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:10:39.423926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:39.423907 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-56c6c9ddd5-67jtl"] Oct 31 00:10:40.079500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:40.079470 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="09669322-f2c4-43b7-9a8e-5285a64f75a1" path="/var/lib/kubelet/pods/09669322-f2c4-43b7-9a8e-5285a64f75a1/volumes" Oct 31 00:10:44.103504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.103479 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:10:44.155921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.155885 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:44.156114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.156078 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" podUID="381200eb-7e51-49ba-a281-2c8117714e85" containerName="maas-api" containerID="cri-o://13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc" gracePeriod=30 Oct 31 00:10:44.397940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.397907 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:44.516476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.516440 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\") pod \"381200eb-7e51-49ba-a281-2c8117714e85\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " Oct 31 00:10:44.516653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.516490 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\") pod \"381200eb-7e51-49ba-a281-2c8117714e85\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " Oct 31 00:10:44.516653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.516527 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\" (UniqueName: \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\") pod \"381200eb-7e51-49ba-a281-2c8117714e85\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " Oct 31 00:10:44.516653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.516633 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\") pod \"381200eb-7e51-49ba-a281-2c8117714e85\" (UID: \"381200eb-7e51-49ba-a281-2c8117714e85\") " Oct 31 00:10:44.517073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.517042 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca" pod "381200eb-7e51-49ba-a281-2c8117714e85" (UID: "381200eb-7e51-49ba-a281-2c8117714e85"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:10:44.518840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.518810 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs" pod "381200eb-7e51-49ba-a281-2c8117714e85" (UID: "381200eb-7e51-49ba-a281-2c8117714e85"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:44.518928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.518851 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls" pod "381200eb-7e51-49ba-a281-2c8117714e85" (UID: "381200eb-7e51-49ba-a281-2c8117714e85"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:10:44.518928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.518863 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd" pod "381200eb-7e51-49ba-a281-2c8117714e85" (UID: "381200eb-7e51-49ba-a281-2c8117714e85"). InnerVolumeSpecName "kube-api-access-4hqvd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:10:44.618101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.618065 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:44.618101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.618097 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4hqvd\" (UniqueName: \"kubernetes.io/projected/381200eb-7e51-49ba-a281-2c8117714e85-kube-api-access-4hqvd\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:44.618101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.618108 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/381200eb-7e51-49ba-a281-2c8117714e85-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:44.618342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:44.618116 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/381200eb-7e51-49ba-a281-2c8117714e85-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:10:45.124818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.124762 6234 generic.go:356] "Generic (PLEG): container finished" podID="381200eb-7e51-49ba-a281-2c8117714e85" containerID="13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc" exitCode=0 Oct 31 00:10:45.125238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.124825 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" Oct 31 00:10:45.125238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.124858 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" event={"ID":"381200eb-7e51-49ba-a281-2c8117714e85","Type":"ContainerDied","Data":"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc"} Oct 31 00:10:45.125238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.124917 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr" event={"ID":"381200eb-7e51-49ba-a281-2c8117714e85","Type":"ContainerDied","Data":"fe2fb7beae55522ccaae206ea1ecc466da6facbb01c44e25f08c6c681eb38398"} Oct 31 00:10:45.125238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.124942 6234 scope.go:122] "RemoveContainer" containerID="13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc" Oct 31 00:10:45.134132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.134113 6234 scope.go:122] "RemoveContainer" containerID="13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc" Oct 31 00:10:45.134386 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:10:45.134360 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc\": container with ID starting with 13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc not found: ID does not exist" containerID="13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc" Oct 31 00:10:45.134441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.134388 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc"} err="failed to get container status \"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc\": rpc error: code = NotFound desc = could not find container \"13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc\": container with ID starting with 13c9683ea08d83a6dd0c437df4d3786ae5c05b58f22b0830e2c1c684d7c844fc not found: ID does not exist" Oct 31 00:10:45.147562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.147533 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:45.152542 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:45.152518 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6d669c7fcd-jvrjr"] Oct 31 00:10:46.079898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:10:46.079865 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="381200eb-7e51-49ba-a281-2c8117714e85" path="/var/lib/kubelet/pods/381200eb-7e51-49ba-a281-2c8117714e85/volumes" Oct 31 00:11:02.997701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.997658 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:11:02.998245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998213 6234 cpu_manager.go:414] "containerMap: removing container" podUID="381200eb-7e51-49ba-a281-2c8117714e85" containerName="maas-api" Oct 31 00:11:02.998245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998238 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="381200eb-7e51-49ba-a281-2c8117714e85" containerName="maas-api" Oct 31 00:11:02.998356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998254 6234 cpu_manager.go:414] "containerMap: removing container" podUID="09669322-f2c4-43b7-9a8e-5285a64f75a1" containerName="authorino" Oct 31 00:11:02.998356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998261 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="09669322-f2c4-43b7-9a8e-5285a64f75a1" containerName="authorino" Oct 31 00:11:02.998453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998355 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="381200eb-7e51-49ba-a281-2c8117714e85" containerName="maas-api" Oct 31 00:11:02.998453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:02.998367 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="09669322-f2c4-43b7-9a8e-5285a64f75a1" containerName="authorino" Oct 31 00:11:03.001869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.001843 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.027007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.026971 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:11:03.060214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060181 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060223 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060246 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4jcqz\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060346 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060388 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060407 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060478 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060560 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.060624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.060610 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.096339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.096305 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p7q4d\"" Oct 31 00:11:03.161408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161370 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161415 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4jcqz\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161449 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:11:03.161451 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161489 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161517 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161558 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161617 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161648 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.161851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161816 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.162001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.161946 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.162047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.162004 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.162085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.162063 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.162241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.162224 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.162323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.162306 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.164247 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.164222 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.165515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.165489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.174508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.174478 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.174874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.174853 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4jcqz\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-wzmnn\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.315625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.315531 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:03.600198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:03.600175 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:11:03.602803 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:03.602754 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20ea1c03_d953_4442_a39e_087b6af667d3.slice/crio-650461026ced6d8cadab4ff2c49a3bca67e0435ffaad5f9ab375b1e8b54ffcdb WatchSource:0}: Error finding container 650461026ced6d8cadab4ff2c49a3bca67e0435ffaad5f9ab375b1e8b54ffcdb: Status 404 returned error can't find the container with id 650461026ced6d8cadab4ff2c49a3bca67e0435ffaad5f9ab375b1e8b54ffcdb Oct 31 00:11:04.204910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:04.204877 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" event={"ID":"20ea1c03-d953-4442-a39e-087b6af667d3","Type":"ContainerStarted","Data":"650461026ced6d8cadab4ff2c49a3bca67e0435ffaad5f9ab375b1e8b54ffcdb"} Oct 31 00:11:07.088233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.088193 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:07.088545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.088286 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:07.088545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.088334 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:07.222607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.222575 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" event={"ID":"20ea1c03-d953-4442-a39e-087b6af667d3","Type":"ContainerStarted","Data":"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009"} Oct 31 00:11:07.250686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.250598 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podStartSLOduration=1.768147133 podStartE2EDuration="5.250582383s" totalImagesPullingTime="3.48243525s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:07.249591167 +0000 UTC m=+989.282687240" watchObservedRunningTime="2026-10-31 00:11:07.250582383 +0000 UTC m=+989.283678456" Oct 31 00:11:07.315850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:07.315817 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:08.317321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:08.317289 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 00:11:08.317699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:08.317348 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 00:11:09.316296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:09.316263 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 00:11:09.316471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:09.316311 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 00:11:10.315798 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:10.315756 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 00:11:10.316179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:10.315837 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 00:11:11.316635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:11.316591 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 00:11:11.317056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:11.316661 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 00:11:12.316235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:12.316194 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body= Oct 31 00:11:12.316444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:12.316256 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" Oct 31 00:11:13.320309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:13.320277 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:13.320688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:13.320639 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:13.321405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:13.321384 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:11:18.036956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.036881 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:11:18.040678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.040653 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.068860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.068833 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:11:18.073350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.073324 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:11:18.077504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.077483 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.106928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.106899 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:11:18.114440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114411 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114467 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114619 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114653 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114740 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114773 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wssh5\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114844 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114902 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.114963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.114920 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216139 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216173 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216201 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216226 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wssh5\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216356 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216407 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.216560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216474 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.216560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216502 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.216669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216624 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.216669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216628 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.216764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216676 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7kzp\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.217025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216720 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.216725 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217411 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217466 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217548 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217710 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217771 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217824 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217844 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.217845 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.218013 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.218128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.218057 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.221535 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.219430 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.224054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.222182 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.229519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.229494 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wssh5\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.230368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.230350 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-8x82m\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.318964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.318886 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.318964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.318930 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.318964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.318947 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.318978 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319023 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319047 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319077 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319206 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7kzp\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319271 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319386 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319454 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319464 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319631 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.319764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.319745 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.321412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.321389 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.321697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.321682 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.343534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.343502 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.343643 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.343542 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7kzp\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-5jxfw\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.353546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.353530 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:18.391190 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.390970 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:18.633602 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.633563 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:11:18.635975 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:18.635949 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5cb52220_22f0_40cf_97c5_a7e9534d4c21.slice/crio-a835fb4819c5dac0f8200a6a6bfaa31d5d0dcbf4fca3e0be6b9eb030705d2f37 WatchSource:0}: Error finding container a835fb4819c5dac0f8200a6a6bfaa31d5d0dcbf4fca3e0be6b9eb030705d2f37: Status 404 returned error can't find the container with id a835fb4819c5dac0f8200a6a6bfaa31d5d0dcbf4fca3e0be6b9eb030705d2f37 Oct 31 00:11:18.637997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.637966 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:18.638071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.638028 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:18.638071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.638056 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:18.680810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.680772 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:11:18.683019 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:18.682993 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf6c8532_d777_42ec_a8ff_11f6e683f0ac.slice/crio-183f8a736f170e597ec0d6e4d5bf99c71dbea0c2c1270e3204be9a0e9a0231d8 WatchSource:0}: Error finding container 183f8a736f170e597ec0d6e4d5bf99c71dbea0c2c1270e3204be9a0e9a0231d8: Status 404 returned error can't find the container with id 183f8a736f170e597ec0d6e4d5bf99c71dbea0c2c1270e3204be9a0e9a0231d8 Oct 31 00:11:18.685306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.685271 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:18.685411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.685350 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:18.685411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:18.685396 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:19.296611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.296573 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" event={"ID":"cf6c8532-d777-42ec-a8ff-11f6e683f0ac","Type":"ContainerStarted","Data":"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f"} Oct 31 00:11:19.297083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.296643 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" event={"ID":"cf6c8532-d777-42ec-a8ff-11f6e683f0ac","Type":"ContainerStarted","Data":"183f8a736f170e597ec0d6e4d5bf99c71dbea0c2c1270e3204be9a0e9a0231d8"} Oct 31 00:11:19.298383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.298361 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" event={"ID":"5cb52220-22f0-40cf-97c5-a7e9534d4c21","Type":"ContainerStarted","Data":"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e"} Oct 31 00:11:19.298501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.298398 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" event={"ID":"5cb52220-22f0-40cf-97c5-a7e9534d4c21","Type":"ContainerStarted","Data":"a835fb4819c5dac0f8200a6a6bfaa31d5d0dcbf4fca3e0be6b9eb030705d2f37"} Oct 31 00:11:19.323025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.322980 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podStartSLOduration=1.3229678169999999 podStartE2EDuration="1.322967817s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:19.32245703 +0000 UTC m=+1001.355553099" watchObservedRunningTime="2026-10-31 00:11:19.322967817 +0000 UTC m=+1001.356063957" Oct 31 00:11:19.359808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.359135 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:19.374809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.360552 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podStartSLOduration=1.360535317 podStartE2EDuration="1.360535317s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:19.359619303 +0000 UTC m=+1001.392715374" watchObservedRunningTime="2026-10-31 00:11:19.360535317 +0000 UTC m=+1001.393631387" Oct 31 00:11:19.374809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.361384 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:19.374809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.361434 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:19.391924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.391398 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:19.392976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.392947 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:19.393127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:19.393003 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:20.356698 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:20.356664 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:20.357315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:20.357286 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:20.391914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:20.391882 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:20.392111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:20.391938 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:21.355278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:21.355239 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:21.355572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:21.355541 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:21.391880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:21.391845 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:21.392363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:21.391917 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:22.358809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:22.355202 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:22.358809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:22.355277 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:22.392372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:22.392327 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:22.392812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:22.392399 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:23.355247 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:23.355208 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:23.355620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:23.355585 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:23.392392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:23.392360 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:23.392874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:23.392542 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:24.361264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:24.361220 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body= Oct 31 00:11:24.361545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:24.361517 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" Oct 31 00:11:24.391472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:24.391432 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body= Oct 31 00:11:24.391672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:24.391485 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" Oct 31 00:11:25.358526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.358491 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:25.358895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.358769 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:25.359405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.359388 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:11:25.395034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.395008 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:25.395285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.395263 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:25.395875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:25.395857 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:11:32.713399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.713368 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp"] Oct 31 00:11:32.716943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.716923 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.722030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.722007 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" Oct 31 00:11:32.737949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.737925 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\"" Oct 31 00:11:32.738096 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.737936 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" Oct 31 00:11:32.741411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.741385 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp"] Oct 31 00:11:32.779203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779164 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t77nh\" (UniqueName: \"kubernetes.io/projected/8d2df4b9-ac3f-474f-a0f6-788c22573929-kube-api-access-t77nh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.779372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779217 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.779372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779321 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.779372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779346 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2df4b9-ac3f-474f-a0f6-788c22573929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.779484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779419 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.779484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.779476 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.822402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.822377 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-wc2gc\"" Oct 31 00:11:32.880934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.880897 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.880971 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t77nh\" (UniqueName: \"kubernetes.io/projected/8d2df4b9-ac3f-474f-a0f6-788c22573929-kube-api-access-t77nh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881010 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881045 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881070 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2df4b9-ac3f-474f-a0f6-788c22573929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881109 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881387 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881445 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.881548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.881525 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.883414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.883381 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8d2df4b9-ac3f-474f-a0f6-788c22573929-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.883766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.883749 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8d2df4b9-ac3f-474f-a0f6-788c22573929-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:32.894612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:32.894582 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t77nh\" (UniqueName: \"kubernetes.io/projected/8d2df4b9-ac3f-474f-a0f6-788c22573929-kube-api-access-t77nh\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp\" (UID: \"8d2df4b9-ac3f-474f-a0f6-788c22573929\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:33.030406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.030311 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:33.083245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.082858 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:11:33.088418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.086684 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.102960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.102877 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:11:33.162062 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:11:33.162020 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:11:33.185269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185235 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185280 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185301 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185416 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185475 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185555 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185574 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.185664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.185588 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7cf2w\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287023 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287075 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287099 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287122 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287223 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7cf2w\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287280 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287342 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287390 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287490 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287554 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287739 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287851 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.287916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.287896 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.289341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.289324 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.289903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.289884 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.309265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.309237 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.309394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.309364 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7cf2w\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-9rmfw\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.313004 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.312984 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp"] Oct 31 00:11:33.315060 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:33.315032 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2df4b9_ac3f_474f_a0f6_788c22573929.slice/crio-518d376f663c4f25e6a1690278d33412444c6bdba3a5c63241f2fe01607266c5 WatchSource:0}: Error finding container 518d376f663c4f25e6a1690278d33412444c6bdba3a5c63241f2fe01607266c5: Status 404 returned error can't find the container with id 518d376f663c4f25e6a1690278d33412444c6bdba3a5c63241f2fe01607266c5 Oct 31 00:11:33.364290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.364256 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" event={"ID":"8d2df4b9-ac3f-474f-a0f6-788c22573929","Type":"ContainerStarted","Data":"518d376f663c4f25e6a1690278d33412444c6bdba3a5c63241f2fe01607266c5"} Oct 31 00:11:33.421327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.421285 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:33.729071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.729031 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:11:33.732898 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:33.732866 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6c51501_e585_422b_88c3_93839313cc78.slice/crio-f95647833d5c2da5abf2bdacc31dd82f9603234f0d5eacbfcaa74f552ffa2f87 WatchSource:0}: Error finding container f95647833d5c2da5abf2bdacc31dd82f9603234f0d5eacbfcaa74f552ffa2f87: Status 404 returned error can't find the container with id f95647833d5c2da5abf2bdacc31dd82f9603234f0d5eacbfcaa74f552ffa2f87 Oct 31 00:11:33.735039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.735007 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:33.735119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.735070 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:33.735119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:33.735096 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:11:34.379868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.379812 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" event={"ID":"c6c51501-e585-422b-88c3-93839313cc78","Type":"ContainerStarted","Data":"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05"} Oct 31 00:11:34.380065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.379920 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" event={"ID":"c6c51501-e585-422b-88c3-93839313cc78","Type":"ContainerStarted","Data":"f95647833d5c2da5abf2bdacc31dd82f9603234f0d5eacbfcaa74f552ffa2f87"} Oct 31 00:11:34.414235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.414182 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podStartSLOduration=1.414168179 podStartE2EDuration="1.414168179s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:34.410894835 +0000 UTC m=+1016.443990906" watchObservedRunningTime="2026-10-31 00:11:34.414168179 +0000 UTC m=+1016.447264248" Oct 31 00:11:34.422008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.421984 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:34.423547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.423524 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:34.423669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:34.423575 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:35.385656 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:35.385620 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" event={"ID":"8d2df4b9-ac3f-474f-a0f6-788c22573929","Type":"ContainerStarted","Data":"83c61d83a142de91fc3073317da7a4cbb23c7bf0a1790fc1ab792a0e4caf4c36"} Oct 31 00:11:35.422154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:35.422115 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:35.422303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:35.422173 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:36.032554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:36.032520 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:36.421906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:36.421820 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:36.421906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:36.421883 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:37.421875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:37.421843 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:37.422068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:37.421892 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:38.425251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:38.425216 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:38.425844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:38.425810 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:39.422208 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.422166 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body= Oct 31 00:11:39.422428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.422223 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" Oct 31 00:11:39.720608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.720553 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" podStartSLOduration=5.85041678 podStartE2EDuration="7.720538651s" totalImagesPullingTime="1.870121871s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:35.449698374 +0000 UTC m=+1017.482794450" watchObservedRunningTime="2026-10-31 00:11:39.720538651 +0000 UTC m=+1021.753634744" Oct 31 00:11:39.722078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.722004 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd"] Oct 31 00:11:39.733092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.733074 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.769848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.741043 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd"] Oct 31 00:11:39.771233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.771202 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\"" Oct 31 00:11:39.816766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.816726 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.816955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.816919 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.817016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.816964 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eccd75-30b9-4a48-88ec-d7aca906afa4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.817016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.816990 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.817117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.817081 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.817166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.817117 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxlqj\" (UniqueName: \"kubernetes.io/projected/76eccd75-30b9-4a48-88ec-d7aca906afa4-kube-api-access-wxlqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918012 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxlqj\" (UniqueName: \"kubernetes.io/projected/76eccd75-30b9-4a48-88ec-d7aca906afa4-kube-api-access-wxlqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918049 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918148 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918184 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eccd75-30b9-4a48-88ec-d7aca906afa4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918212 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918251 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918582 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-home\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918657 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-model-cache\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.918696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.918683 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-tmp-dir\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.921045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.921023 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76eccd75-30b9-4a48-88ec-d7aca906afa4-dshm\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.921531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.921501 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76eccd75-30b9-4a48-88ec-d7aca906afa4-tls-certs\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:39.931384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:39.931364 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxlqj\" (UniqueName: \"kubernetes.io/projected/76eccd75-30b9-4a48-88ec-d7aca906afa4-kube-api-access-wxlqj\") pod \"e2e-distinct-simulated-kserve-7b998b79d9-7mjdd\" (UID: \"76eccd75-30b9-4a48-88ec-d7aca906afa4\") " pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:40.079193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:40.079115 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:40.356369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:40.356346 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd"] Oct 31 00:11:40.358635 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:40.358597 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76eccd75_30b9_4a48_88ec_d7aca906afa4.slice/crio-3dd60df0ed9cabd3126acfdca6c9dd1577598dec3c9be231da14d8dcb3733714 WatchSource:0}: Error finding container 3dd60df0ed9cabd3126acfdca6c9dd1577598dec3c9be231da14d8dcb3733714: Status 404 returned error can't find the container with id 3dd60df0ed9cabd3126acfdca6c9dd1577598dec3c9be231da14d8dcb3733714 Oct 31 00:11:40.436369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:40.436336 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" event={"ID":"76eccd75-30b9-4a48-88ec-d7aca906afa4","Type":"ContainerStarted","Data":"3dd60df0ed9cabd3126acfdca6c9dd1577598dec3c9be231da14d8dcb3733714"} Oct 31 00:11:41.422295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.422209 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:41.422844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.422302 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:41.456829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.452707 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" event={"ID":"76eccd75-30b9-4a48-88ec-d7aca906afa4","Type":"ContainerStarted","Data":"1236f516760e064f480c7617e9efbec7d4a8c0a8bf935331517f9a1f3d25e8de"} Oct 31 00:11:41.486701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.486634 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" podStartSLOduration=2.17612916 podStartE2EDuration="2.486615356s" totalImagesPullingTime="310.486196ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:41.485420143 +0000 UTC m=+1023.518516213" watchObservedRunningTime="2026-10-31 00:11:41.486615356 +0000 UTC m=+1023.519711427" Oct 31 00:11:41.500005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.499975 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8"] Oct 31 00:11:41.587144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.587102 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8"] Oct 31 00:11:41.587545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.587527 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.607807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.604099 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639125 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639184 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639215 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639290 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959c5b03-4058-432e-8769-91e97e5deed9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639400 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhr47\" (UniqueName: \"kubernetes.io/projected/959c5b03-4058-432e-8769-91e97e5deed9-kube-api-access-jhr47\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.644972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.639428 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742370 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhr47\" (UniqueName: \"kubernetes.io/projected/959c5b03-4058-432e-8769-91e97e5deed9-kube-api-access-jhr47\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742421 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742466 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742506 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742536 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.744812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.742609 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959c5b03-4058-432e-8769-91e97e5deed9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.757230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.757198 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.789815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.779139 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.789815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.779467 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.818963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.818923 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/959c5b03-4058-432e-8769-91e97e5deed9-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.819432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.819400 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/959c5b03-4058-432e-8769-91e97e5deed9-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:41.867846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:41.860715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhr47\" (UniqueName: \"kubernetes.io/projected/959c5b03-4058-432e-8769-91e97e5deed9-kube-api-access-jhr47\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-r99g8\" (UID: \"959c5b03-4058-432e-8769-91e97e5deed9\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:42.006315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:42.005989 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:42.116419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:42.116388 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:42.423820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:42.423710 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:42.423820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:42.423807 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:42.485794 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:42.485734 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8"] Oct 31 00:11:42.493430 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:11:42.493401 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod959c5b03_4058_432e_8769_91e97e5deed9.slice/crio-0e15490bc7214b8f1d9506872b5e5d8e035201474c3074fdfa6fbb5e815203e9 WatchSource:0}: Error finding container 0e15490bc7214b8f1d9506872b5e5d8e035201474c3074fdfa6fbb5e815203e9: Status 404 returned error can't find the container with id 0e15490bc7214b8f1d9506872b5e5d8e035201474c3074fdfa6fbb5e815203e9 Oct 31 00:11:43.030951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.030835 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:43.051207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.051174 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:43.465957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.465922 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:11:43.466675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.466644 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:11:43.472288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.471751 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" event={"ID":"959c5b03-4058-432e-8769-91e97e5deed9","Type":"ContainerStarted","Data":"3f488135da7d79a6935efbf09c388c22d77725e1cb5204d93480dddfe73bbc27"} Oct 31 00:11:43.472288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.471845 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" event={"ID":"959c5b03-4058-432e-8769-91e97e5deed9","Type":"ContainerStarted","Data":"0e15490bc7214b8f1d9506872b5e5d8e035201474c3074fdfa6fbb5e815203e9"} Oct 31 00:11:43.491526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.491504 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" Oct 31 00:11:43.518817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.518510 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" podStartSLOduration=2.24002599 podStartE2EDuration="2.518488412s" totalImagesPullingTime="278.462422ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:43.503542692 +0000 UTC m=+1025.536638766" watchObservedRunningTime="2026-10-31 00:11:43.518488412 +0000 UTC m=+1025.551584484" Oct 31 00:11:43.768866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.768192 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:43.769519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.769496 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:43.789872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:43.789833 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:11:44.017817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:44.016895 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:46.211676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.211636 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bfgv"] Oct 31 00:11:46.216771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.216742 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.238974 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"echo-simulator-kserve-self-signed-certs\"" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240298 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240364 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240398 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240438 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240517 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8s2r\" (UniqueName: \"kubernetes.io/projected/354612ef-bb88-4c36-b5fe-caf0ca868ce7-kube-api-access-l8s2r\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240551 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.242461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.240924 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bfgv"] Oct 31 00:11:46.346365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.346324 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.346644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.346617 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.346799 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.346761 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.346962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.346942 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.347167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.347148 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8s2r\" (UniqueName: \"kubernetes.io/projected/354612ef-bb88-4c36-b5fe-caf0ca868ce7-kube-api-access-l8s2r\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.347287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.347271 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.359656 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.359618 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-home\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.360464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.360433 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-model-cache\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.362298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.362240 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tmp-dir\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.363348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.363176 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/354612ef-bb88-4c36-b5fe-caf0ca868ce7-dshm\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.364396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.364373 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/354612ef-bb88-4c36-b5fe-caf0ca868ce7-tls-certs\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.376305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.376283 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8s2r\" (UniqueName: \"kubernetes.io/projected/354612ef-bb88-4c36-b5fe-caf0ca868ce7-kube-api-access-l8s2r\") pod \"echo-simulator-kserve-64b6f8f96c-5bfgv\" (UID: \"354612ef-bb88-4c36-b5fe-caf0ca868ce7\") " pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.552082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.551969 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:46.856405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:46.855520 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/echo-simulator-kserve-64b6f8f96c-5bfgv"] Oct 31 00:11:47.581592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:47.581495 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" event={"ID":"354612ef-bb88-4c36-b5fe-caf0ca868ce7","Type":"ContainerStarted","Data":"1d8b1b34202bf12c3c5455dfcbe75f8f981ca9d2f63916c7729966ccda49f5c6"} Oct 31 00:11:50.089826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:50.088422 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:50.102108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:50.102021 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:50.650528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:50.650493 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" event={"ID":"354612ef-bb88-4c36-b5fe-caf0ca868ce7","Type":"ContainerStarted","Data":"71b1320834e81ff2a4fbdbfae04b219b7ab14e54ae6fcc76b11ffad992012c6a"} Oct 31 00:11:50.688394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:50.688341 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" podStartSLOduration=1.706660431 podStartE2EDuration="4.688322182s" totalImagesPullingTime="2.981661751s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:11:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:11:50.685330679 +0000 UTC m=+1032.718426751" watchObservedRunningTime="2026-10-31 00:11:50.688322182 +0000 UTC m=+1032.721418253" Oct 31 00:11:50.700095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:50.700069 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" Oct 31 00:11:51.556809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:51.554880 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:52.009726 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:52.009695 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:52.036095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:52.036062 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:52.672272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:52.672247 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-r99g8" Oct 31 00:11:56.553083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:56.553044 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:56.565636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:56.565608 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:11:56.691681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:11:56.691646 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" Oct 31 00:12:03.161012 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:12:03.160968 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:12:04.477555 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.477516 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k94mw"] Oct 31 00:12:04.481479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.481458 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.492816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.492046 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k94mw"] Oct 31 00:12:04.559225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.556424 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\"" Oct 31 00:12:04.559225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.556603 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-pxrsw\"" Oct 31 00:12:04.560997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.560904 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.561279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.561257 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rzkr8\" (UniqueName: \"kubernetes.io/projected/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-kube-api-access-rzkr8\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.663186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.663157 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rzkr8\" (UniqueName: \"kubernetes.io/projected/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-kube-api-access-rzkr8\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.663393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.663245 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.663929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.663909 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-plugins-config-volume\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.676921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.676894 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rzkr8\" (UniqueName: \"kubernetes.io/projected/8e683085-dae1-46d7-8e0f-b04bdd08a7dc-kube-api-access-rzkr8\") pod \"payload-processing-55fbbbf9dc-k94mw\" (UID: \"8e683085-dae1-46d7-8e0f-b04bdd08a7dc\") " pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:04.798117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:04.798030 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:05.077235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:05.077210 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-55fbbbf9dc-k94mw"] Oct 31 00:12:05.078608 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:12:05.078582 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e683085_dae1_46d7_8e0f_b04bdd08a7dc.slice/crio-0aa6f814c04857b2210db50676e3af1a8e372ddb2077b6ac701a1507e0b46061 WatchSource:0}: Error finding container 0aa6f814c04857b2210db50676e3af1a8e372ddb2077b6ac701a1507e0b46061: Status 404 returned error can't find the container with id 0aa6f814c04857b2210db50676e3af1a8e372ddb2077b6ac701a1507e0b46061 Oct 31 00:12:05.724360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:05.724323 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" event={"ID":"8e683085-dae1-46d7-8e0f-b04bdd08a7dc","Type":"ContainerStarted","Data":"0aa6f814c04857b2210db50676e3af1a8e372ddb2077b6ac701a1507e0b46061"} Oct 31 00:12:07.736560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:07.736522 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" event={"ID":"8e683085-dae1-46d7-8e0f-b04bdd08a7dc","Type":"ContainerStarted","Data":"4e460062cc3c48973a57a57d1171a9afdf1610e34983655f1f0f96a4eb7a04b3"} Oct 31 00:12:07.736992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:07.736630 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:07.756490 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:07.756442 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" podStartSLOduration=1.306164901 podStartE2EDuration="3.75642547s" totalImagesPullingTime="2.450260569s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:12:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:12:07.756354494 +0000 UTC m=+1049.789450764" watchObservedRunningTime="2026-10-31 00:12:07.75642547 +0000 UTC m=+1049.789521541" Oct 31 00:12:18.744098 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:12:18.744063 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-55fbbbf9dc-k94mw" Oct 31 00:12:33.162007 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:12:33.161970 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:13:02.172907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.172807 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:13:02.176455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.176430 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.192214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.192177 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-7b2341-openshift-default-dockercfg-7tk5j\"" Oct 31 00:13:02.196261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.196233 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:13:02.336821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.336772 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.336994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.336885 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.336994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.336924 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.336994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.336963 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.337159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.337019 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.337159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.337048 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.337159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.337106 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.337159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.337131 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.337159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.337151 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf42w\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438137 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438179 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438201 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf42w\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438233 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438258 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438285 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438330 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438447 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438499 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438686 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.438770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438748 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.439018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.438819 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.439082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.439059 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.439506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.439484 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.441283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.441256 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.442872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.442848 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.454231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.454203 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf42w\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.456692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.456669 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.490276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.490245 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:02.511975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.511939 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:13:02.516599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.516548 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.527678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.527646 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-4c360e-openshift-default-dockercfg-22rlx\"" Oct 31 00:13:02.535866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.535752 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:13:02.541058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.541016 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.541432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.541343 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.541691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.541614 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.541980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.541959 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.543451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.543429 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.543640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.543623 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.543801 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.543771 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.543923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.543906 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.544012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.543945 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p5tbx\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645004 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645056 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645090 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p5tbx\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645194 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645416 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645438 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645469 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645504 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645528 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645574 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645593 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.645978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.645946 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.646144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.646102 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.646211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.646186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.648348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.648324 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.648587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.648569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.658148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.658115 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.661638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.661613 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p5tbx\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.728648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.728574 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:13:02.733245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.733226 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.741395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.741370 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-0402df-openshift-default-dockercfg-q78hj\"" Oct 31 00:13:02.746467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.746419 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.746467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.746465 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.747996 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-krxlw\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748058 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748166 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748242 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748303 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748334 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.748392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.748369 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.749837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.749813 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:13:02.794073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.794041 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:13:02.794655 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:02.794621 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9bf53c8_a0b5_44ed_bffe_a2aca9b48b5c.slice/crio-e92cdfd32e78a4eaed54dbe9b0a62e8f39221129c04da50231abcb74bc3b640a WatchSource:0}: Error finding container e92cdfd32e78a4eaed54dbe9b0a62e8f39221129c04da50231abcb74bc3b640a: Status 404 returned error can't find the container with id e92cdfd32e78a4eaed54dbe9b0a62e8f39221129c04da50231abcb74bc3b640a Oct 31 00:13:02.801474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.801435 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:02.801713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.801526 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:02.801830 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.801747 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:02.835245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.835211 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:02.849615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849575 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.849847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849648 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.849847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849683 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.849847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849795 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.849847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849826 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849907 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-krxlw\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849952 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.849980 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850017 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850043 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850347 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850841 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.850974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850923 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.851039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.850998 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.853279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.853246 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.853384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.853332 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.871851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.871818 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-krxlw\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:02.872095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:02.872070 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:03.006859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.006367 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerStarted","Data":"47a153db7ef4906f494be395424f3eba88ac73925e7dde12d137e8c8818240c3"} Oct 31 00:13:03.006859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.006439 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerStarted","Data":"e92cdfd32e78a4eaed54dbe9b0a62e8f39221129c04da50231abcb74bc3b640a"} Oct 31 00:13:03.049994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.049675 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:03.133860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.133279 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podStartSLOduration=1.133254986 podStartE2EDuration="1.133254986s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:03.049471143 +0000 UTC m=+1105.082567216" watchObservedRunningTime="2026-10-31 00:13:03.133254986 +0000 UTC m=+1105.166351067" Oct 31 00:13:03.133860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.133637 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:13:03.137852 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:03.137820 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb993a36b_71fe_4361_a86d_fd8e6b4bd396.slice/crio-f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126 WatchSource:0}: Error finding container f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126: Status 404 returned error can't find the container with id f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126 Oct 31 00:13:03.140980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.140715 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.140980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.140777 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.140980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.140865 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.206272 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:03.206128 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:13:03.346407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.346355 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:13:03.349282 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:03.349253 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode268a382_a45c_48c9_adc5_cb3c7b117920.slice/crio-9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9 WatchSource:0}: Error finding container 9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9: Status 404 returned error can't find the container with id 9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9 Oct 31 00:13:03.351670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.351636 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.351764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.351705 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.351764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.351733 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:03.491330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.491293 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:03.497649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:03.497621 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:04.013338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.013298 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerStarted","Data":"6ba28c750492180b363d7baaaf2a98d9826621b7ca6408c9955c01bd50b7bb0b"} Oct 31 00:13:04.013550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.013366 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerStarted","Data":"f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126"} Oct 31 00:13:04.014946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.014919 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerStarted","Data":"6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02"} Oct 31 00:13:04.015071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.014995 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:04.015071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.015025 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerStarted","Data":"9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9"} Oct 31 00:13:04.015853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.015773 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:04.040084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.040028 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podStartSLOduration=2.040014242 podStartE2EDuration="2.040014242s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:04.038462193 +0000 UTC m=+1106.071558264" watchObservedRunningTime="2026-10-31 00:13:04.040014242 +0000 UTC m=+1106.073110311" Oct 31 00:13:04.051371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.051328 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:04.056113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.056089 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:04.062254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.062215 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podStartSLOduration=2.062199046 podStartE2EDuration="2.062199046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:04.061971251 +0000 UTC m=+1106.095067346" watchObservedRunningTime="2026-10-31 00:13:04.062199046 +0000 UTC m=+1106.095295115" Oct 31 00:13:04.835699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.835663 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:04.840909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:04.840884 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:05.020133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:05.020013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:05.020133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:05.020094 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:05.021058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:05.021035 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:05.021159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:05.021089 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:07.584606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.584550 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:13:07.588386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.588370 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.596280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.596254 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:13:07.596405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.596383 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w5-7b2341\"" Oct 31 00:13:07.599647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.599630 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.606195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.606177 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w5-7b2341-dockercfg-gsp5r\"" Oct 31 00:13:07.616860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.616836 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:13:07.618826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.618807 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:13:07.711946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.711908 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vhjq7\" (UniqueName: \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.712146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.711995 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.712146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.712061 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9b6nb\" (UniqueName: \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.712146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.712097 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.813416 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.813383 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.813600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.813484 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vhjq7\" (UniqueName: \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.813600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.813543 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.813600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.813594 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9b6nb\" (UniqueName: \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.814091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.814068 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.814226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.814209 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.826616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.826586 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vhjq7\" (UniqueName: \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\") pod \"payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.826867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.826848 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9b6nb\" (UniqueName: \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\") pod \"payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:07.902016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.901932 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:07.911149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:07.911129 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:08.184674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.184648 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:13:08.186432 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:08.186402 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2467d0c4_552c_4916_9b15_ac1aaea4c5c1.slice/crio-f2d1fe77d98e421f42f5f368f0d77b55f3f4f914396686786268af8e2d4eaf52 WatchSource:0}: Error finding container f2d1fe77d98e421f42f5f368f0d77b55f3f4f914396686786268af8e2d4eaf52: Status 404 returned error can't find the container with id f2d1fe77d98e421f42f5f368f0d77b55f3f4f914396686786268af8e2d4eaf52 Oct 31 00:13:08.204974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.204948 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:13:08.213955 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:08.213930 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda29a19b4_12de_464e_8b28_9f1598267fea.slice/crio-28980a58f7cfc2ba3f48154941fb1626ac511d71c3f41f33411ea7be3394ce93 WatchSource:0}: Error finding container 28980a58f7cfc2ba3f48154941fb1626ac511d71c3f41f33411ea7be3394ce93: Status 404 returned error can't find the container with id 28980a58f7cfc2ba3f48154941fb1626ac511d71c3f41f33411ea7be3394ce93 Oct 31 00:13:08.509754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.509704 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:13:08.513383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.513364 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.524223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.524197 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:13:08.589185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.589149 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-7c5da3\"" Oct 31 00:13:08.589687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.589327 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-7c5da3-dockercfg-sxrt5\"" Oct 31 00:13:08.621527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.621482 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.621728 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.621612 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x84x5\" (UniqueName: \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.722402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.722372 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x84x5\" (UniqueName: \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.722585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.722478 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.723241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.723221 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.739317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.739296 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x84x5\" (UniqueName: \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\") pod \"payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.767187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.767132 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:13:08.773191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.773162 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.788699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.788660 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:13:08.809972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.809938 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w5-7b2341-serving-cert\"" Oct 31 00:13:08.823510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.823477 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.823644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.823518 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.823644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.823538 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r5mmk\" (UniqueName: \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.823832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.823771 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.827049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.827027 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:08.924427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.924383 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.924427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.924427 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.924670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.924456 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r5mmk\" (UniqueName: \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.924670 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:08.924512 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-serving-cert: secret "maas-api-e2e-worker-w5-7b2341-serving-cert" not found Oct 31 00:13:08.924670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.924563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.924670 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:08.924584 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls podName:2fc57b11-8551-469c-92b3-611656e2615c nodeName:}" failed. No retries permitted until 2026-10-31 00:13:09.424562677 +0000 UTC m=+1111.457658745 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls") pod "maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" (UID: "2fc57b11-8551-469c-92b3-611656e2615c") : secret "maas-api-e2e-worker-w5-7b2341-serving-cert" not found Oct 31 00:13:08.925435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.925413 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.928234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.928210 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:08.945276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:08.945253 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r5mmk\" (UniqueName: \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:09.041524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.041431 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" event={"ID":"a29a19b4-12de-464e-8b28-9f1598267fea","Type":"ContainerStarted","Data":"71c7c409d8af9b2040339853ecd07605c77541f3626418fc8510c609dd5af533"} Oct 31 00:13:09.041524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.041488 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" event={"ID":"a29a19b4-12de-464e-8b28-9f1598267fea","Type":"ContainerStarted","Data":"28980a58f7cfc2ba3f48154941fb1626ac511d71c3f41f33411ea7be3394ce93"} Oct 31 00:13:09.041524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.041517 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:09.043065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.043041 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" event={"ID":"2467d0c4-552c-4916-9b15-ac1aaea4c5c1","Type":"ContainerStarted","Data":"48bfe8bdb01b104a477e99969892f3e89f82a019a6b047cabcaf5af1963336dc"} Oct 31 00:13:09.043211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.043077 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" event={"ID":"2467d0c4-552c-4916-9b15-ac1aaea4c5c1","Type":"ContainerStarted","Data":"f2d1fe77d98e421f42f5f368f0d77b55f3f4f914396686786268af8e2d4eaf52"} Oct 31 00:13:09.043211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.043164 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:09.098093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.098036 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" podStartSLOduration=2.098020218 podStartE2EDuration="2.098020218s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:09.09682066 +0000 UTC m=+1111.129916722" watchObservedRunningTime="2026-10-31 00:13:09.098020218 +0000 UTC m=+1111.131116289" Oct 31 00:13:09.098536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.098508 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" podStartSLOduration=2.098501236 podStartE2EDuration="2.098501236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:09.070917761 +0000 UTC m=+1111.104013836" watchObservedRunningTime="2026-10-31 00:13:09.098501236 +0000 UTC m=+1111.131597305" Oct 31 00:13:09.115316 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.115287 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:13:09.116390 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:09.116364 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f0be845_0d78_4855_99e8_fceee5d1db1e.slice/crio-7faf50e13602c647f4e58386d8414da138f6bceb1a026ce3bfb7a0cff5ebdc47 WatchSource:0}: Error finding container 7faf50e13602c647f4e58386d8414da138f6bceb1a026ce3bfb7a0cff5ebdc47: Status 404 returned error can't find the container with id 7faf50e13602c647f4e58386d8414da138f6bceb1a026ce3bfb7a0cff5ebdc47 Oct 31 00:13:09.445479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.445440 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:09.448071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.448041 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") pod \"maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:09.690399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.690364 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:09.997814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:09.997696 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:13:10.000769 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:10.000739 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2fc57b11_8551_469c_92b3_611656e2615c.slice/crio-3b233f95ab1ad1031ad8c6f96b4f773670018016018257a0be0f327e6731f9ac WatchSource:0}: Error finding container 3b233f95ab1ad1031ad8c6f96b4f773670018016018257a0be0f327e6731f9ac: Status 404 returned error can't find the container with id 3b233f95ab1ad1031ad8c6f96b4f773670018016018257a0be0f327e6731f9ac Oct 31 00:13:10.050093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.050061 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" event={"ID":"6f0be845-0d78-4855-99e8-fceee5d1db1e","Type":"ContainerStarted","Data":"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456"} Oct 31 00:13:10.050253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.050112 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" event={"ID":"6f0be845-0d78-4855-99e8-fceee5d1db1e","Type":"ContainerStarted","Data":"7faf50e13602c647f4e58386d8414da138f6bceb1a026ce3bfb7a0cff5ebdc47"} Oct 31 00:13:10.050253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.050141 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:10.051348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.051320 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" event={"ID":"2fc57b11-8551-469c-92b3-611656e2615c","Type":"ContainerStarted","Data":"3b233f95ab1ad1031ad8c6f96b4f773670018016018257a0be0f327e6731f9ac"} Oct 31 00:13:10.082193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.082133 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" podStartSLOduration=2.082114708 podStartE2EDuration="2.082114708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:10.080660433 +0000 UTC m=+1112.113756503" watchObservedRunningTime="2026-10-31 00:13:10.082114708 +0000 UTC m=+1112.115210781" Oct 31 00:13:10.130500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.130464 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:13:10.135295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.135280 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.146167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.145659 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:13:10.149604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.149582 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-d91de6\"" Oct 31 00:13:10.263091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.263018 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgmxs\" (UniqueName: \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.263256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.263236 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.365758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.365715 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.367089 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.367065 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgmxs\" (UniqueName: \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.367663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.366946 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.384930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.384895 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgmxs\" (UniqueName: \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\") pod \"payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.450002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.449967 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:10.734624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:10.734601 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:13:10.736497 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:10.736470 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode07e4d29_303a_4c97_8715_e9804db8d803.slice/crio-76c708489d048b4ab5b920de894760378d16d93e1da3e69ad39cd14fbd173cce WatchSource:0}: Error finding container 76c708489d048b4ab5b920de894760378d16d93e1da3e69ad39cd14fbd173cce: Status 404 returned error can't find the container with id 76c708489d048b4ab5b920de894760378d16d93e1da3e69ad39cd14fbd173cce Oct 31 00:13:11.078864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.078538 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" event={"ID":"2fc57b11-8551-469c-92b3-611656e2615c","Type":"ContainerStarted","Data":"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f"} Oct 31 00:13:11.082838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.079340 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:11.082838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.081530 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" event={"ID":"e07e4d29-303a-4c97-8715-e9804db8d803","Type":"ContainerStarted","Data":"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24"} Oct 31 00:13:11.082838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.081565 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" event={"ID":"e07e4d29-303a-4c97-8715-e9804db8d803","Type":"ContainerStarted","Data":"76c708489d048b4ab5b920de894760378d16d93e1da3e69ad39cd14fbd173cce"} Oct 31 00:13:11.082838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.081585 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:11.116935 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.116880 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" podStartSLOduration=2.652906714 podStartE2EDuration="3.116864865s" totalImagesPullingTime="463.958151ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.115755314 +0000 UTC m=+1113.148851386" watchObservedRunningTime="2026-10-31 00:13:11.116864865 +0000 UTC m=+1113.149960935" Oct 31 00:13:11.138907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.138853 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" podStartSLOduration=1.138838667 podStartE2EDuration="1.138838667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:11.13729807 +0000 UTC m=+1113.170394143" watchObservedRunningTime="2026-10-31 00:13:11.138838667 +0000 UTC m=+1113.171934759" Oct 31 00:13:11.495076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.495042 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:13:11.499011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.498988 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.509052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.508980 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w6-0402df-dockercfg-m2sz5\"" Oct 31 00:13:11.509641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.509618 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w6-0402df\"" Oct 31 00:13:11.511584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.511564 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:13:11.586044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.586013 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.586222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.586134 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kljxh\" (UniqueName: \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.687611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.687559 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kljxh\" (UniqueName: \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.687925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.687747 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.688675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.688647 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.732975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.732941 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kljxh\" (UniqueName: \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\") pod \"payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:11.813369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:11.813275 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:12.132696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.132479 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:13:12.136512 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:12.136482 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf169e7e9_2bd0_4b9c_bd7a_ce62509ac47f.slice/crio-97f68b538cfbdb56a0523b39f2a2698ed9b94f3583da9ec57fa6dc733476af53 WatchSource:0}: Error finding container 97f68b538cfbdb56a0523b39f2a2698ed9b94f3583da9ec57fa6dc733476af53: Status 404 returned error can't find the container with id 97f68b538cfbdb56a0523b39f2a2698ed9b94f3583da9ec57fa6dc733476af53 Oct 31 00:13:12.409625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.409551 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:13:12.413304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.413285 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.425091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.425064 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w1-4c360e\"" Oct 31 00:13:12.426348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.426325 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:13:12.471829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.471798 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:13:12.475891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.475870 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.481233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.481199 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w1-4c360e-dockercfg-sqzzm\"" Oct 31 00:13:12.487250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.487221 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:13:12.496336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.496299 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ld764\" (UniqueName: \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.496517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.496499 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.597687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.597643 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.598049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.598017 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-98jjq\" (UniqueName: \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.598189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.598110 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.598244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.598212 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ld764\" (UniqueName: \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.599144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.599117 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.618750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.618713 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ld764\" (UniqueName: \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\") pod \"payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.699681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.699644 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.699893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.699690 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-98jjq\" (UniqueName: \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.700325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.700304 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.712347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.712322 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-98jjq\" (UniqueName: \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\") pod \"payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.727222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.727199 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:12.769714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.769672 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:13:12.774756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.774731 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:12.781284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.781257 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:13:12.781520 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.781501 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w0-c60826-serving-cert\"" Oct 31 00:13:12.791453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.791430 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:12.905659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.904726 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:12.905659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.904818 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:12.905659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.904966 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:12.905659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:12.905006 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vrbd\" (UniqueName: \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.006203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.006106 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.006357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.006233 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.006357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.006277 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vrbd\" (UniqueName: \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.006469 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.006361 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.006533 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:13.006510 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-serving-cert: secret "maas-api-e2e-worker-w0-c60826-serving-cert" not found Oct 31 00:13:13.006607 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:13.006592 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls podName:7b85dda1-ff52-42e7-b9fd-f54f6fcb803f nodeName:}" failed. No retries permitted until 2026-10-31 00:13:13.50656967 +0000 UTC m=+1115.539665739 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls") pod "maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" (UID: "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f") : secret "maas-api-e2e-worker-w0-c60826-serving-cert" not found Oct 31 00:13:13.007064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.007047 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.008820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.008803 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.020967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.020940 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vrbd\" (UniqueName: \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.068356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.068253 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:13:13.069256 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:13.069209 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1f38638_649c_42ae_afac_6eb04bfbdd91.slice/crio-8a890a49981472787b1fb4e9d793b82dec0b89f2651d58a54648e193e55358e1 WatchSource:0}: Error finding container 8a890a49981472787b1fb4e9d793b82dec0b89f2651d58a54648e193e55358e1: Status 404 returned error can't find the container with id 8a890a49981472787b1fb4e9d793b82dec0b89f2651d58a54648e193e55358e1 Oct 31 00:13:13.091934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.091895 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" event={"ID":"f1f38638-649c-42ae-afac-6eb04bfbdd91","Type":"ContainerStarted","Data":"8a890a49981472787b1fb4e9d793b82dec0b89f2651d58a54648e193e55358e1"} Oct 31 00:13:13.093703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.093677 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" event={"ID":"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f","Type":"ContainerStarted","Data":"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a"} Oct 31 00:13:13.093828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.093727 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" event={"ID":"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f","Type":"ContainerStarted","Data":"97f68b538cfbdb56a0523b39f2a2698ed9b94f3583da9ec57fa6dc733476af53"} Oct 31 00:13:13.094056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.094038 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:13.116955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.116913 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" podStartSLOduration=2.116899447 podStartE2EDuration="2.116899447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:13.114395324 +0000 UTC m=+1115.147491393" watchObservedRunningTime="2026-10-31 00:13:13.116899447 +0000 UTC m=+1115.149995517" Oct 31 00:13:13.125633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.125567 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:13:13.128122 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:13.128088 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad4bbafa_37e8_403d_84a6_6c4437086688.slice/crio-271f393f8037b8cf38fb5b6fd41f3f651cf294e02d0ca08c3cd2dc203513403e WatchSource:0}: Error finding container 271f393f8037b8cf38fb5b6fd41f3f651cf294e02d0ca08c3cd2dc203513403e: Status 404 returned error can't find the container with id 271f393f8037b8cf38fb5b6fd41f3f651cf294e02d0ca08c3cd2dc203513403e Oct 31 00:13:13.521305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.521264 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.528451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.528385 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") pod \"maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:13.693445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:13.693408 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:14.057059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.053511 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:13:14.100238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.100196 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" event={"ID":"ad4bbafa-37e8-403d-84a6-6c4437086688","Type":"ContainerStarted","Data":"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c"} Oct 31 00:13:14.100371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.100262 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" event={"ID":"ad4bbafa-37e8-403d-84a6-6c4437086688","Type":"ContainerStarted","Data":"271f393f8037b8cf38fb5b6fd41f3f651cf294e02d0ca08c3cd2dc203513403e"} Oct 31 00:13:14.100732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.100442 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:14.102236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.102133 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" event={"ID":"f1f38638-649c-42ae-afac-6eb04bfbdd91","Type":"ContainerStarted","Data":"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d"} Oct 31 00:13:14.102437 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.102249 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:14.103858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.103840 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" event={"ID":"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f","Type":"ContainerStarted","Data":"321bdd3f2489b56e6863fbd5c89c2aa6bbf7ab9b05b3fa3b0ed9f05b03120b9e"} Oct 31 00:13:14.122925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.122883 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" podStartSLOduration=2.122870535 podStartE2EDuration="2.122870535s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:14.121063041 +0000 UTC m=+1116.154159138" watchObservedRunningTime="2026-10-31 00:13:14.122870535 +0000 UTC m=+1116.155966604" Oct 31 00:13:14.160983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:14.155975 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" podStartSLOduration=2.155956191 podStartE2EDuration="2.155956191s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:14.154223082 +0000 UTC m=+1116.187319153" watchObservedRunningTime="2026-10-31 00:13:14.155956191 +0000 UTC m=+1116.189052261" Oct 31 00:13:15.111658 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:15.111619 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" event={"ID":"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f","Type":"ContainerStarted","Data":"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71"} Oct 31 00:13:15.112205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:15.112186 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:15.134258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:15.134200 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" podStartSLOduration=2.669248885 podStartE2EDuration="3.134186716s" totalImagesPullingTime="464.937831ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:15.132652166 +0000 UTC m=+1117.165748237" watchObservedRunningTime="2026-10-31 00:13:15.134186716 +0000 UTC m=+1117.167282785" Oct 31 00:13:16.770609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.770565 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:13:16.775149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.775129 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:16.788507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.788481 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w6-0402df-serving-cert\"" Oct 31 00:13:16.793920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.793890 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:13:16.906806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.899946 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:16.906806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.900055 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:16.906806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.900093 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t6dqq\" (UniqueName: \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:16.906806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:16.900245 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.001522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.001485 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.001692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.001534 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.001692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.001574 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.001692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.001591 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t6dqq\" (UniqueName: \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.002183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.002162 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.004266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.004242 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.004710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.004694 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.015608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.015584 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t6dqq\" (UniqueName: \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\") pod \"maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.090127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.090057 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:13:17.090258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.090172 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:17.410446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:17.410409 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:13:18.131594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:18.131536 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" event={"ID":"0f21f026-cd45-4d72-a915-b1d87334b488","Type":"ContainerStarted","Data":"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21"} Oct 31 00:13:18.132009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:18.131685 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:18.132009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:18.131705 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" event={"ID":"0f21f026-cd45-4d72-a915-b1d87334b488","Type":"ContainerStarted","Data":"76dd25ab044ee5f4535d2501aea93d48aff0790f401708708ad2608baf23f995"} Oct 31 00:13:18.156723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:18.156653 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" podStartSLOduration=1.645058064 podStartE2EDuration="2.156634058s" totalImagesPullingTime="511.575994ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:18.15398819 +0000 UTC m=+1120.187084286" watchObservedRunningTime="2026-10-31 00:13:18.156634058 +0000 UTC m=+1120.189730130" Oct 31 00:13:20.054402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:20.054374 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:13:20.054880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:20.054448 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:13:20.056929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:20.056904 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:13:20.057036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:20.056956 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:13:21.084647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:21.084615 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:13:22.088240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.088211 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:13:22.175596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.175564 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:13:22.184742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.184695 6234 generic.go:356] "Generic (PLEG): container finished" podID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerID="47a153db7ef4906f494be395424f3eba88ac73925e7dde12d137e8c8818240c3" exitCode=0 Oct 31 00:13:22.184969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.184814 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerDied","Data":"47a153db7ef4906f494be395424f3eba88ac73925e7dde12d137e8c8818240c3"} Oct 31 00:13:22.186881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.185255 6234 scope.go:122] "RemoveContainer" containerID="47a153db7ef4906f494be395424f3eba88ac73925e7dde12d137e8c8818240c3" Oct 31 00:13:22.187807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.187552 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:22.187807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.187717 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:22.187962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:22.187836 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:23.194021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:23.193990 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerStarted","Data":"a59651bf71f42f4ef4e0b0c90a7266522edc271235de1c07584edda77efeff0e"} Oct 31 00:13:23.500881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:23.500251 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:23.500881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:23.500394 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:13:23.500881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:23.500449 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:13:24.107349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:24.107319 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:13:24.192001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:24.191969 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:13:24.491362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:24.491251 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:13:24.491362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:24.491320 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:13:25.115173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:25.115143 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:13:25.115362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:25.115208 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:13:25.491254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:25.491216 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:13:25.491528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:25.491503 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:13:26.490682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:26.490640 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:13:26.490853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:26.490697 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:13:27.490986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:27.490951 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" start-of-body= Oct 31 00:13:27.491361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:27.491002 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": dial tcp 10.134.0.44:15021: connect: connection refused" Oct 31 00:13:28.827639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:28.827539 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:28.828452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:28.828423 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:28.830177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:28.830154 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:13:31.303933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.303902 6234 generic.go:356] "Generic (PLEG): container finished" podID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerID="6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02" exitCode=0 Oct 31 00:13:31.304430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.303939 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerDied","Data":"6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02"} Oct 31 00:13:31.304430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.304312 6234 scope.go:122] "RemoveContainer" containerID="6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02" Oct 31 00:13:31.305592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.305563 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:31.305654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.305629 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:31.305687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:31.305664 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:32.315072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:32.315036 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerStarted","Data":"16f8100ac391b1af1f4a85accb81c423d998483561f57b3938612fda336fd21b"} Oct 31 00:13:33.053441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:33.053404 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:33.053831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:33.053806 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:13:33.054018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:33.053994 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:13:33.205044 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:13:33.205005 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:13:34.060849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:34.057271 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:13:34.060849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:34.057341 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:13:35.020768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:35.020727 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:35.058818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:35.058194 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:13:35.058818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:35.058295 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:13:36.063515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:36.063143 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:13:36.063515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:36.063213 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:13:36.064085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:36.063880 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:13:36.064085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:36.063934 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:13:37.064328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.064290 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" start-of-body= Oct 31 00:13:37.064850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.064361 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.46:15021/healthz/ready\": dial tcp 10.134.0.46:15021: connect: connection refused" Oct 31 00:13:37.374757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.374726 6234 generic.go:356] "Generic (PLEG): container finished" podID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerID="6ba28c750492180b363d7baaaf2a98d9826621b7ca6408c9955c01bd50b7bb0b" exitCode=0 Oct 31 00:13:37.374966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.374821 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerDied","Data":"6ba28c750492180b363d7baaaf2a98d9826621b7ca6408c9955c01bd50b7bb0b"} Oct 31 00:13:37.375261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.375244 6234 scope.go:122] "RemoveContainer" containerID="6ba28c750492180b363d7baaaf2a98d9826621b7ca6408c9955c01bd50b7bb0b" Oct 31 00:13:37.376811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.376764 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:37.376894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.376852 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:37.376894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:37.376883 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:13:38.055900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.055871 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:38.057584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.057562 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:13:38.382109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.382015 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerStarted","Data":"fb8112cc159faaf807e0854ed42adacaab68be4661f3c0680696f911a21c045c"} Oct 31 00:13:38.836085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.836048 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:38.836289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.836262 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:13:38.836389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:38.836334 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:13:39.836590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:39.836506 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:13:39.837223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:39.837185 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:13:40.093396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.093326 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:13:40.099254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.099232 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.105624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.105579 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:13:40.107606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.107583 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"kube-root-ca.crt\"" Oct 31 00:13:40.107606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.107583 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"simulator-w3-19adb9-kserve-self-signed-certs\"" Oct 31 00:13:40.108964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.108946 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"openshift-service-ca.crt\"" Oct 31 00:13:40.109062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.108964 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-19adb9\"/\"default-dockercfg-8kfxl\"" Oct 31 00:13:40.186478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186438 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.186651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186499 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.186651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186552 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.186651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186581 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.186752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186666 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.186752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.186707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-745zz\" (UniqueName: \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.288663 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.288802 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.288847 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.288895 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.288995 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.289855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.289022 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-745zz\" (UniqueName: \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.290676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.290652 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.291177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.291116 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.291604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.291582 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.297826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.296941 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.311303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.310661 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-745zz\" (UniqueName: \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.311303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.311242 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\") pod \"simulator-w3-19adb9-kserve-5d557b7984-6rd2k\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.413650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.413566 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:40.741210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.739572 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:13:40.836351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.836322 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:13:40.836478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:40.836367 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:13:41.400734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.400698 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" event={"ID":"8769c5e0-af3a-4da6-ae9e-f316e967ce48","Type":"ContainerStarted","Data":"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68"} Oct 31 00:13:41.401186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.400746 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" event={"ID":"8769c5e0-af3a-4da6-ae9e-f316e967ce48","Type":"ContainerStarted","Data":"3e05ed2d239838aa47876e3365dde9ebe59998a5d6ccc5a2a9ff218011dd708d"} Oct 31 00:13:41.414074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.414049 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:41.437994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.429192 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podStartSLOduration=1.429177766 podStartE2EDuration="1.429177766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:41.426977118 +0000 UTC m=+1143.460073201" watchObservedRunningTime="2026-10-31 00:13:41.429177766 +0000 UTC m=+1143.462273836" Oct 31 00:13:41.836139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.836108 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:13:41.836335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:41.836163 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:13:42.835735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:42.835699 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" start-of-body= Oct 31 00:13:42.836153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:42.835751 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": dial tcp 10.134.0.45:15021: connect: connection refused" Oct 31 00:13:44.012859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:44.012823 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:44.014700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:44.013615 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:44.014700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:44.014373 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:13:48.598112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.598080 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:13:48.603086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.603069 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.615233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.615208 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"openshift-service-ca.crt\"" Oct 31 00:13:48.615487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.615387 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"default-dockercfg-r8fdg\"" Oct 31 00:13:48.615487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.615483 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"kube-root-ca.crt\"" Oct 31 00:13:48.616042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.616021 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:13:48.616568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.616551 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"simulator-w0-c60826-kserve-self-signed-certs\"" Oct 31 00:13:48.686210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686176 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rmml6\" (UniqueName: \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.686371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686243 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.686371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686332 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.686371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686362 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.686478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686459 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.686511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.686494 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.801189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.801152 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.802069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.802038 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.802286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.802268 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.802446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.802433 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.802538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.802526 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.802746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.802727 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rmml6\" (UniqueName: \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.803297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.801933 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.806426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.805860 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.809162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.806893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.812413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.812359 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.814347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.813981 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.817269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.817220 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rmml6\" (UniqueName: \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\") pod \"simulator-w0-c60826-kserve-5d6dd5b68-6pnk2\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:48.917759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:48.917675 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:49.213293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:49.213271 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:13:49.215362 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:13:49.215333 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod719d7203_6520_42c5_9c07_0837c6561a8d.slice/crio-35605ffe6d8851d580f98e0eef00657f84aced78389afa9a749e4ad56357a7c2 WatchSource:0}: Error finding container 35605ffe6d8851d580f98e0eef00657f84aced78389afa9a749e4ad56357a7c2: Status 404 returned error can't find the container with id 35605ffe6d8851d580f98e0eef00657f84aced78389afa9a749e4ad56357a7c2 Oct 31 00:13:49.447867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:49.447830 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" event={"ID":"719d7203-6520-42c5-9c07-0837c6561a8d","Type":"ContainerStarted","Data":"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9"} Oct 31 00:13:49.448038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:49.447880 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" event={"ID":"719d7203-6520-42c5-9c07-0837c6561a8d","Type":"ContainerStarted","Data":"35605ffe6d8851d580f98e0eef00657f84aced78389afa9a749e4ad56357a7c2"} Oct 31 00:13:49.471262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:49.471174 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podStartSLOduration=1.47116117 podStartE2EDuration="1.47116117s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:13:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:13:49.470121046 +0000 UTC m=+1151.503217117" watchObservedRunningTime="2026-10-31 00:13:49.47116117 +0000 UTC m=+1151.504257240" Oct 31 00:13:49.918824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:49.918717 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:50.414690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:50.414648 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:50.427445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:50.427414 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:50.464401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:50.464374 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:13:58.918615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:58.918579 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:58.931439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:58.931414 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:13:59.575865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:13:59.575833 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:14:03.161668 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:14:03.161629 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:14:09.070667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.070636 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:14:09.074897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.074870 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.089367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.089339 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"kube-root-ca.crt\"" Oct 31 00:14:09.089367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.089371 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"openshift-service-ca.crt\"" Oct 31 00:14:09.089534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.089384 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"default-dockercfg-h6gq5\"" Oct 31 00:14:09.089534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.089413 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"premium-w2-7c5da3-kserve-self-signed-certs\"" Oct 31 00:14:09.090330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.090313 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:14:09.185243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185209 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.185415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185265 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v6866\" (UniqueName: \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.185415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185373 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.185536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185484 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.185536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185525 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.185653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.185637 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287090 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287052 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287123 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287188 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v6866\" (UniqueName: \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287257 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287332 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287361 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287557 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287643 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287621 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.287755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.287738 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.289741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.289711 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.289982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.289966 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.316032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.316011 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v6866\" (UniqueName: \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\") pod \"premium-w2-7c5da3-kserve-5bbb77fb6f-njksx\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.387978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.387904 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:09.669675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:09.669647 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:14:09.671732 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:14:09.671700 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb1516214_0b59_4ab0_ae3e_eb4aad219009.slice/crio-b31495dc7567f12b109c203945c82b070c1eca25788d25b1ce53885252027675 WatchSource:0}: Error finding container b31495dc7567f12b109c203945c82b070c1eca25788d25b1ce53885252027675: Status 404 returned error can't find the container with id b31495dc7567f12b109c203945c82b070c1eca25788d25b1ce53885252027675 Oct 31 00:14:10.612429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:10.612390 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" event={"ID":"b1516214-0b59-4ab0-ae3e-eb4aad219009","Type":"ContainerStarted","Data":"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c"} Oct 31 00:14:10.612429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:10.612435 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" event={"ID":"b1516214-0b59-4ab0-ae3e-eb4aad219009","Type":"ContainerStarted","Data":"b31495dc7567f12b109c203945c82b070c1eca25788d25b1ce53885252027675"} Oct 31 00:14:10.636161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:10.636111 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podStartSLOduration=1.636097658 podStartE2EDuration="1.636097658s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:10.63440816 +0000 UTC m=+1172.667504231" watchObservedRunningTime="2026-10-31 00:14:10.636097658 +0000 UTC m=+1172.669193727" Oct 31 00:14:11.388368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:11.388329 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:15.871913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.871878 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:14:15.876968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.876947 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:15.888114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.888090 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"kube-root-ca.crt\"" Oct 31 00:14:15.888237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.888089 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"openshift-service-ca.crt\"" Oct 31 00:14:15.888411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.888391 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:14:15.889163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.889147 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"default-dockercfg-lpfcw\"" Oct 31 00:14:15.889213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:15.889169 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"premium-w6-0402df-kserve-self-signed-certs\"" Oct 31 00:14:16.054806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.054755 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.054995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.054827 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.054995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.054973 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.055085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.055032 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.055119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.055094 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8sf9g\" (UniqueName: \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.055153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.055127 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156379 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156431 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156467 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156492 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156514 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8sf9g\" (UniqueName: \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156612 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156888 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.156990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.156934 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.157048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.157021 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.158908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.158885 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.159176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.159158 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.171747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.171721 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8sf9g\" (UniqueName: \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\") pod \"premium-w6-0402df-kserve-8d7957496-l48qv\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.191940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.191911 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:16.472410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.472380 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:14:16.474367 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:14:16.474331 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3c13ba68_b7ce_40f4_8232_c8d2203cd6b4.slice/crio-0f1f0c8eca1e020b09014b5eb41cb8c8a622b5970714914e30b9bc8b97ab23a2 WatchSource:0}: Error finding container 0f1f0c8eca1e020b09014b5eb41cb8c8a622b5970714914e30b9bc8b97ab23a2: Status 404 returned error can't find the container with id 0f1f0c8eca1e020b09014b5eb41cb8c8a622b5970714914e30b9bc8b97ab23a2 Oct 31 00:14:16.669567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.667297 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" event={"ID":"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4","Type":"ContainerStarted","Data":"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c"} Oct 31 00:14:16.669567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.667413 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" event={"ID":"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4","Type":"ContainerStarted","Data":"0f1f0c8eca1e020b09014b5eb41cb8c8a622b5970714914e30b9bc8b97ab23a2"} Oct 31 00:14:16.713856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:16.713772 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podStartSLOduration=1.713757422 podStartE2EDuration="1.713757422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:16.71252978 +0000 UTC m=+1178.745625851" watchObservedRunningTime="2026-10-31 00:14:16.713757422 +0000 UTC m=+1178.746853529" Oct 31 00:14:17.193073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.193036 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:17.462110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.462033 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:14:17.465982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.465961 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.470581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.470556 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-7c5da3-openshift-default-dockercfg-87p8b\"" Oct 31 00:14:17.477669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477606 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.477777 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477667 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.477864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477795 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.477864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477844 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pckhm\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.477985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477962 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.478022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.477998 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.478069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.478055 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.478144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.478110 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.478198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.478180 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.488525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.488500 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:14:17.579093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579059 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579106 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579138 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579159 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pckhm\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579209 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579225 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579275 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579300 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579332 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579494 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579585 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579767 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.579955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.579875 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.580073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.580056 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.581533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.581513 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.581850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.581835 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.608466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.607280 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pckhm\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.624283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.624238 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:17.781381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:17.781296 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:18.068684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.068661 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:14:18.070166 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:14:18.070140 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod801ed45b_760b_456a_ad10_178de4ed72e4.slice/crio-897752837e3075c6390d6fafee08b57502840c880498d3273eb33ce2486f6a87 WatchSource:0}: Error finding container 897752837e3075c6390d6fafee08b57502840c880498d3273eb33ce2486f6a87: Status 404 returned error can't find the container with id 897752837e3075c6390d6fafee08b57502840c880498d3273eb33ce2486f6a87 Oct 31 00:14:18.072621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.072581 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:18.072706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.072649 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:18.072706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.072679 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:18.689754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.689715 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" event={"ID":"801ed45b-760b-456a-ad10-178de4ed72e4","Type":"ContainerStarted","Data":"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951"} Oct 31 00:14:18.690208 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.689815 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" event={"ID":"801ed45b-760b-456a-ad10-178de4ed72e4","Type":"ContainerStarted","Data":"897752837e3075c6390d6fafee08b57502840c880498d3273eb33ce2486f6a87"} Oct 31 00:14:18.724423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.724374 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podStartSLOduration=1.7243608369999999 podStartE2EDuration="1.724360837s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:18.722020344 +0000 UTC m=+1180.755116428" watchObservedRunningTime="2026-10-31 00:14:18.724360837 +0000 UTC m=+1180.757456907" Oct 31 00:14:18.781561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:18.781523 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:19.395055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:19.392929 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:19.425817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:19.424829 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:19.707188 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:19.707159 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:14:19.782763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:19.782725 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:14:19.782945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:19.782832 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:14:20.782133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:20.782096 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:14:20.782527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:20.782163 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:14:21.781718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:21.781685 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:14:21.781887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:21.781737 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:14:22.782196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:22.782148 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:14:22.782719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:22.782218 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:14:23.782468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:23.782433 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" start-of-body= Oct 31 00:14:23.782899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:23.782501 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": dial tcp 10.134.0.61:15021: connect: connection refused" Oct 31 00:14:24.829219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:24.829187 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:24.829719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:24.829694 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:24.831249 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:24.831222 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:14:26.192280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:26.192241 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:26.204842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:26.204819 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:26.749987 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:26.749953 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:14:32.994453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:32.994414 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:14:32.998630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:32.998607 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.009479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.009455 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-19adb9-openshift-default-dockercfg-csw9z\"" Oct 31 00:14:33.015538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.015511 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:14:33.114520 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114478 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114520 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114522 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114554 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114667 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114708 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sch67\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114825 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.114917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114896 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.115010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114928 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.115010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.114947 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.162422 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:14:33.162385 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:14:33.215939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.215893 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.215942 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sch67\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216218 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216249 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216351 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216418 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216446 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216476 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216746 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216908 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.216953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216928 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.217055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.216956 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.218409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.218387 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.218973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.218955 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.229353 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.229332 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.229492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.229474 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sch67\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.312290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.312203 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:33.599692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.599664 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:14:33.601156 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:14:33.601125 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode6b0be86_0695_4bfd_a065_e395378e4a66.slice/crio-7bb57eeff5ed21958cb7eccd473b82569f61d1f70dc0a0cddc74234965eba312 WatchSource:0}: Error finding container 7bb57eeff5ed21958cb7eccd473b82569f61d1f70dc0a0cddc74234965eba312: Status 404 returned error can't find the container with id 7bb57eeff5ed21958cb7eccd473b82569f61d1f70dc0a0cddc74234965eba312 Oct 31 00:14:33.603506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.603473 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:33.603559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.603540 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:33.603590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.603575 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:14:33.779774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.779742 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" event={"ID":"e6b0be86-0695-4bfd-a065-e395378e4a66","Type":"ContainerStarted","Data":"87db2d0dc6c18319f1337697452d99e498a779f578639f74076aa1308fa3194a"} Oct 31 00:14:33.779954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.779816 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" event={"ID":"e6b0be86-0695-4bfd-a065-e395378e4a66","Type":"ContainerStarted","Data":"7bb57eeff5ed21958cb7eccd473b82569f61d1f70dc0a0cddc74234965eba312"} Oct 31 00:14:33.807237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:33.807177 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podStartSLOduration=1.8071581079999999 podStartE2EDuration="1.807158108s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:14:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:14:33.804191272 +0000 UTC m=+1195.837287341" watchObservedRunningTime="2026-10-31 00:14:33.807158108 +0000 UTC m=+1195.840254179" Oct 31 00:14:34.312687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:34.312647 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:34.314227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:34.314204 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:34.314316 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:34.314258 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:35.313075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:35.313038 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:35.313664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:35.313090 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:36.312915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:36.312877 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:36.313101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:36.312931 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:37.313067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:37.313035 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:37.313271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:37.313103 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:38.313079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:38.313041 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:38.313273 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:38.313129 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:39.312713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:39.312681 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" start-of-body= Oct 31 00:14:39.312904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:39.312738 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": dial tcp 10.134.0.62:15021: connect: connection refused" Oct 31 00:14:40.512214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:40.512153 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:40.513084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:40.513061 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:14:40.514144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:14:40.514118 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:15:00.193582 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.193548 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:00.197526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.197505 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.240887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.240855 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:00.244970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.244944 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.293068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.293034 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:00.308380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.308352 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:15:00.317767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.317729 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.318025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.317822 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9qb5\" (UniqueName: \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.318025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.317909 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:00.419081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.419047 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.419258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.419108 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2bhw8\" (UniqueName: \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.419258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.419208 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.419258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.419251 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9qb5\" (UniqueName: \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.419884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.419860 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.433211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.433186 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9qb5\" (UniqueName: \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\") pod \"maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.512769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.512682 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:00.520555 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.520528 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.520625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.520616 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2bhw8\" (UniqueName: \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.521088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.521070 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.536125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.536098 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2bhw8\" (UniqueName: \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\") pod \"maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.558736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.558703 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:00.797538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.797510 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:00.798977 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:15:00.798951 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8715a1e_a353_45d9_9fb8_9379e88acd68.slice/crio-8a7ce86e1f4798d7bbed790a23ea7bb32f0444e3b867f118af92f8baf958003e WatchSource:0}: Error finding container 8a7ce86e1f4798d7bbed790a23ea7bb32f0444e3b867f118af92f8baf958003e: Status 404 returned error can't find the container with id 8a7ce86e1f4798d7bbed790a23ea7bb32f0444e3b867f118af92f8baf958003e Oct 31 00:15:00.800883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.800868 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:15:00.872764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.872738 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:00.873891 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:15:00.873853 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee96b161_f9ba_48d7_8c6e_9d8efdc63c59.slice/crio-bbd7ac6c28c796a61dfb38b08c63e1aef6398a31eadd277b054870a815f7a6f6 WatchSource:0}: Error finding container bbd7ac6c28c796a61dfb38b08c63e1aef6398a31eadd277b054870a815f7a6f6: Status 404 returned error can't find the container with id bbd7ac6c28c796a61dfb38b08c63e1aef6398a31eadd277b054870a815f7a6f6 Oct 31 00:15:00.971576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.971540 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerStarted","Data":"bbd7ac6c28c796a61dfb38b08c63e1aef6398a31eadd277b054870a815f7a6f6"} Oct 31 00:15:00.972767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:00.972741 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" event={"ID":"f8715a1e-a353-45d9-9fb8-9379e88acd68","Type":"ContainerStarted","Data":"8a7ce86e1f4798d7bbed790a23ea7bb32f0444e3b867f118af92f8baf958003e"} Oct 31 00:15:03.169253 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:03.169169 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:15:05.003419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:05.003374 6234 generic.go:356] "Generic (PLEG): container finished" podID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerID="6b61110e66f18678f0987a73d7e6ab948c30e238db541549efe0c4cd7bb2107b" exitCode=60 Oct 31 00:15:05.003917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:05.003438 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" event={"ID":"f8715a1e-a353-45d9-9fb8-9379e88acd68","Type":"ContainerDied","Data":"6b61110e66f18678f0987a73d7e6ab948c30e238db541549efe0c4cd7bb2107b"} Oct 31 00:15:05.003917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:05.003898 6234 scope.go:122] "RemoveContainer" containerID="6b61110e66f18678f0987a73d7e6ab948c30e238db541549efe0c4cd7bb2107b" Oct 31 00:15:05.006725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:05.006691 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerStarted","Data":"cd1cddc385d35f45c565318108a5ec44ad6d390566f85e0cb11e7f40f165964a"} Oct 31 00:15:05.050612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:05.050101 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" podStartSLOduration=1.515581426 podStartE2EDuration="5.050080464s" totalImagesPullingTime="3.534499038s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:05.046196687 +0000 UTC m=+1227.079292767" watchObservedRunningTime="2026-10-31 00:15:05.050080464 +0000 UTC m=+1227.083176536" Oct 31 00:15:06.014358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.014325 6234 generic.go:356] "Generic (PLEG): container finished" podID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerID="cd1cddc385d35f45c565318108a5ec44ad6d390566f85e0cb11e7f40f165964a" exitCode=60 Oct 31 00:15:06.014828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.014401 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerDied","Data":"cd1cddc385d35f45c565318108a5ec44ad6d390566f85e0cb11e7f40f165964a"} Oct 31 00:15:06.014828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.014724 6234 scope.go:122] "RemoveContainer" containerID="cd1cddc385d35f45c565318108a5ec44ad6d390566f85e0cb11e7f40f165964a" Oct 31 00:15:06.016253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.016232 6234 generic.go:356] "Generic (PLEG): container finished" podID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerID="71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f" exitCode=60 Oct 31 00:15:06.016371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.016356 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" event={"ID":"f8715a1e-a353-45d9-9fb8-9379e88acd68","Type":"ContainerDied","Data":"71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f"} Oct 31 00:15:06.016423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.016408 6234 scope.go:122] "RemoveContainer" containerID="6b61110e66f18678f0987a73d7e6ab948c30e238db541549efe0c4cd7bb2107b" Oct 31 00:15:06.016526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:06.016510 6234 scope.go:122] "RemoveContainer" containerID="71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f" Oct 31 00:15:06.016759 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:06.016737 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt_odh-ai-gateway-infra(f8715a1e-a353-45d9-9fb8-9379e88acd68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" Oct 31 00:15:07.024237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:07.024203 6234 generic.go:356] "Generic (PLEG): container finished" podID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerID="24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d" exitCode=60 Oct 31 00:15:07.024748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:07.024292 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerDied","Data":"24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d"} Oct 31 00:15:07.024748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:07.024355 6234 scope.go:122] "RemoveContainer" containerID="cd1cddc385d35f45c565318108a5ec44ad6d390566f85e0cb11e7f40f165964a" Oct 31 00:15:07.024748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:07.024667 6234 scope.go:122] "RemoveContainer" containerID="24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d" Oct 31 00:15:07.025034 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:07.025014 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h_odh-ai-gateway-infra(ee96b161-f9ba-48d7-8c6e-9d8efdc63c59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" Oct 31 00:15:07.026352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:07.026336 6234 scope.go:122] "RemoveContainer" containerID="71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f" Oct 31 00:15:07.026559 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:07.026540 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt_odh-ai-gateway-infra(f8715a1e-a353-45d9-9fb8-9379e88acd68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" Oct 31 00:15:08.033837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:08.033773 6234 scope.go:122] "RemoveContainer" containerID="24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d" Oct 31 00:15:08.034220 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:08.034034 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h_odh-ai-gateway-infra(ee96b161-f9ba-48d7-8c6e-9d8efdc63c59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" Oct 31 00:15:17.881636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.874862 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:15:17.899615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.899579 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:15:17.899818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.899722 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.905592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.905563 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-c60826-openshift-default-dockercfg-v9d6d\"" Oct 31 00:15:17.976769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.976723 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.976991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.976826 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.976991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.976921 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.976988 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.977089 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crzkd\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.977136 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.977171 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.977240 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:17.977303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:17.977293 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.078483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078446 6234 scope.go:122] "RemoveContainer" containerID="71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f" Oct 31 00:15:18.078731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078708 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.078808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078744 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.078852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078806 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.078902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078869 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.078971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078953 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crzkd\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.078996 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079025 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079076 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079125 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079145 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079157 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079746 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.079997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.079975 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.080251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.080234 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.081973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.081948 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.082072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.082007 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.096001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.095967 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.096157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.096066 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crzkd\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.218632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.218594 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:18.518949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.518921 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:15:18.521648 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:15:18.521620 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4abcb2ca_7779_4387_b0af_1579b448227b.slice/crio-f56f6a371370738f410174a1357a306ab6f15d7efa3aedc64a803f3cec467b53 WatchSource:0}: Error finding container f56f6a371370738f410174a1357a306ab6f15d7efa3aedc64a803f3cec467b53: Status 404 returned error can't find the container with id f56f6a371370738f410174a1357a306ab6f15d7efa3aedc64a803f3cec467b53 Oct 31 00:15:18.524270 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.524227 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:15:18.524360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.524347 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:15:18.524413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:18.524388 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:15:19.107346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.107310 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" event={"ID":"4abcb2ca-7779-4387-b0af-1579b448227b","Type":"ContainerStarted","Data":"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7"} Oct 31 00:15:19.107848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.107375 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" event={"ID":"4abcb2ca-7779-4387-b0af-1579b448227b","Type":"ContainerStarted","Data":"f56f6a371370738f410174a1357a306ab6f15d7efa3aedc64a803f3cec467b53"} Oct 31 00:15:19.109149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.109126 6234 generic.go:356] "Generic (PLEG): container finished" podID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerID="75b9148924db77f26aae66c1d51a9018b095466bc9663d5252598a598db09ddc" exitCode=60 Oct 31 00:15:19.109260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.109195 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" event={"ID":"f8715a1e-a353-45d9-9fb8-9379e88acd68","Type":"ContainerDied","Data":"75b9148924db77f26aae66c1d51a9018b095466bc9663d5252598a598db09ddc"} Oct 31 00:15:19.109260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.109246 6234 scope.go:122] "RemoveContainer" containerID="71f89850e65eec540dd8686f16912a6399ca28306d24c60517e33e2a435b9a5f" Oct 31 00:15:19.109605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.109587 6234 scope.go:122] "RemoveContainer" containerID="75b9148924db77f26aae66c1d51a9018b095466bc9663d5252598a598db09ddc" Oct 31 00:15:19.109934 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:19.109911 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt_odh-ai-gateway-infra(f8715a1e-a353-45d9-9fb8-9379e88acd68)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" Oct 31 00:15:19.136219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.136155 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podStartSLOduration=2.136141527 podStartE2EDuration="2.136141527s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:19.132829116 +0000 UTC m=+1241.165925184" watchObservedRunningTime="2026-10-31 00:15:19.136141527 +0000 UTC m=+1241.169237657" Oct 31 00:15:19.219842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.219746 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:19.222099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.222066 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:19.222242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:19.222124 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:20.166873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.166823 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:20.219855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.219819 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:20.220056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.219895 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:20.391319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.391008 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:20.525121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.525090 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\") pod \"f8715a1e-a353-45d9-9fb8-9379e88acd68\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " Oct 31 00:15:20.525289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.525192 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\" (UniqueName: \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\") pod \"f8715a1e-a353-45d9-9fb8-9379e88acd68\" (UID: \"f8715a1e-a353-45d9-9fb8-9379e88acd68\") " Oct 31 00:15:20.525434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.525412 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca" pod "f8715a1e-a353-45d9-9fb8-9379e88acd68" (UID: "f8715a1e-a353-45d9-9fb8-9379e88acd68"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:20.525773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.525754 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f8715a1e-a353-45d9-9fb8-9379e88acd68-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:20.527472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.527445 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5" pod "f8715a1e-a353-45d9-9fb8-9379e88acd68" (UID: "f8715a1e-a353-45d9-9fb8-9379e88acd68"). InnerVolumeSpecName "kube-api-access-z9qb5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:20.627129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:20.627090 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9qb5\" (UniqueName: \"kubernetes.io/projected/f8715a1e-a353-45d9-9fb8-9379e88acd68-kube-api-access-z9qb5\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:21.085817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.075446 6234 scope.go:122] "RemoveContainer" containerID="24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d" Oct 31 00:15:21.127749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.127721 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" Oct 31 00:15:21.128010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.127957 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt" event={"ID":"f8715a1e-a353-45d9-9fb8-9379e88acd68","Type":"ContainerDied","Data":"8a7ce86e1f4798d7bbed790a23ea7bb32f0444e3b867f118af92f8baf958003e"} Oct 31 00:15:21.128102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.128032 6234 scope.go:122] "RemoveContainer" containerID="75b9148924db77f26aae66c1d51a9018b095466bc9663d5252598a598db09ddc" Oct 31 00:15:21.196485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.196452 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:21.203841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.203812 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-d91de6-29890095-8c5gt"] Oct 31 00:15:21.219797 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.219749 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:21.219940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:21.219831 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:22.088809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:22.088348 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" path="/var/lib/kubelet/pods/f8715a1e-a353-45d9-9fb8-9379e88acd68/volumes" Oct 31 00:15:22.136501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:22.136465 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerStarted","Data":"1d1eb54ec586b89a29f03b479b6d0a5fe7fcc321570e44fa374bdb61a725564e"} Oct 31 00:15:22.219382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:22.219346 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:22.219812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:22.219401 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:23.171606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.170390 6234 generic.go:356] "Generic (PLEG): container finished" podID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerID="1d1eb54ec586b89a29f03b479b6d0a5fe7fcc321570e44fa374bdb61a725564e" exitCode=60 Oct 31 00:15:23.173614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.173551 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerDied","Data":"1d1eb54ec586b89a29f03b479b6d0a5fe7fcc321570e44fa374bdb61a725564e"} Oct 31 00:15:23.174304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.174241 6234 scope.go:122] "RemoveContainer" containerID="24da3f924f3323a26b3472d4b6f302362f15fcb594e3896466fc3081ef289c5d" Oct 31 00:15:23.201086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.201014 6234 scope.go:122] "RemoveContainer" containerID="1d1eb54ec586b89a29f03b479b6d0a5fe7fcc321570e44fa374bdb61a725564e" Oct 31 00:15:23.202532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.202503 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:23.214794 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:23.211857 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h_odh-ai-gateway-infra(ee96b161-f9ba-48d7-8c6e-9d8efdc63c59)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" Oct 31 00:15:23.219235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.219203 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:23.219369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:23.219269 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:24.219948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.219918 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:24.220328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.219967 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:24.390600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.390571 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:24.417990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.417957 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\") pod \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " Oct 31 00:15:24.418187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.418156 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\" (UniqueName: \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\") pod \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\" (UID: \"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59\") " Oct 31 00:15:24.419161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.419134 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca" pod "ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" (UID: "ee96b161-f9ba-48d7-8c6e-9d8efdc63c59"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:15:24.421938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.421905 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8" pod "ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" (UID: "ee96b161-f9ba-48d7-8c6e-9d8efdc63c59"). InnerVolumeSpecName "kube-api-access-2bhw8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:15:24.520045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.520017 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2bhw8\" (UniqueName: \"kubernetes.io/projected/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-kube-api-access-2bhw8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:24.520045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:24.520043 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:15:25.189744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.189711 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" Oct 31 00:15:25.189965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.189708 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h" event={"ID":"ee96b161-f9ba-48d7-8c6e-9d8efdc63c59","Type":"ContainerDied","Data":"bbd7ac6c28c796a61dfb38b08c63e1aef6398a31eadd277b054870a815f7a6f6"} Oct 31 00:15:25.189965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.189886 6234 scope.go:122] "RemoveContainer" containerID="1d1eb54ec586b89a29f03b479b6d0a5fe7fcc321570e44fa374bdb61a725564e" Oct 31 00:15:25.215194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.215157 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:25.219328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.219303 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body= Oct 31 00:15:25.219466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.219351 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" Oct 31 00:15:25.222767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:25.222731 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-c60826-29890095-5fm9h"] Oct 31 00:15:26.082451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:26.082410 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" path="/var/lib/kubelet/pods/ee96b161-f9ba-48d7-8c6e-9d8efdc63c59/volumes" Oct 31 00:15:26.223963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:26.223935 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:26.224341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:26.224296 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:26.225092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:26.225073 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:15:33.308307 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:15:33.308235 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:15:41.812241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812208 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:15:41.812808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812793 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812809 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812821 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812826 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812842 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812850 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812862 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.812866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812867 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812892 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812896 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.812991 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.813006 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.813016 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.813023 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.813029 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8715a1e-a353-45d9-9fb8-9379e88acd68" containerName="cleanup" Oct 31 00:15:41.813069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.813036 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:15:41.820198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.820175 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.832689 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.832663 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"default-dockercfg-v7xzz\"" Oct 31 00:15:41.832927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.832697 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"openshift-service-ca.crt\"" Oct 31 00:15:41.832927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.832697 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"distinct-w4-d91de6-kserve-self-signed-certs\"" Oct 31 00:15:41.833044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.832815 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-d91de6\"/\"kube-root-ca.crt\"" Oct 31 00:15:41.835231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.835208 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:15:41.921346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921311 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.921504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921370 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.921548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921507 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.921583 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921545 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.921583 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921563 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lnxgr\" (UniqueName: \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:41.921645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:41.921628 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.022723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022688 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.022922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022746 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.022922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022841 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.022922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022868 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.022922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022892 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lnxgr\" (UniqueName: \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.023121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.022929 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.023295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.023270 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.023295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.023288 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.023460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.023327 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.025247 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.025225 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.025566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.025547 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.036299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.036275 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lnxgr\" (UniqueName: \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\") pod \"distinct-w4-d91de6-kserve-545d4d87b-sbzw2\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.134627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.134549 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:42.414162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:42.414131 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:15:42.415355 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:15:42.415330 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0d148c2b_1b9f_4782_acbe_42c5b821a395.slice/crio-d1ab59922f91b9304f836610c820612afdd3040921f305a2a10d89a7a97bbb79 WatchSource:0}: Error finding container d1ab59922f91b9304f836610c820612afdd3040921f305a2a10d89a7a97bbb79: Status 404 returned error can't find the container with id d1ab59922f91b9304f836610c820612afdd3040921f305a2a10d89a7a97bbb79 Oct 31 00:15:43.392430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:43.392393 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" event={"ID":"0d148c2b-1b9f-4782-acbe-42c5b821a395","Type":"ContainerStarted","Data":"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847"} Oct 31 00:15:43.392430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:43.392436 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" event={"ID":"0d148c2b-1b9f-4782-acbe-42c5b821a395","Type":"ContainerStarted","Data":"d1ab59922f91b9304f836610c820612afdd3040921f305a2a10d89a7a97bbb79"} Oct 31 00:15:43.417043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:43.416984 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podStartSLOduration=2.41696744 podStartE2EDuration="2.41696744s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:15:43.416161208 +0000 UTC m=+1265.449257275" watchObservedRunningTime="2026-10-31 00:15:43.41696744 +0000 UTC m=+1265.450063511" Oct 31 00:15:44.134918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:44.134887 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:52.134982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:52.134897 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:52.147508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:52.147482 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:15:52.560075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:15:52.560048 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:16:03.146596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.146466 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:16:03.160010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.159967 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:16:03.160245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.160228 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee96b161-f9ba-48d7-8c6e-9d8efdc63c59" containerName="cleanup" Oct 31 00:16:03.203938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.203903 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:16:03.204267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.204251 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.264758 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:16:03.264704 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:16:03.301128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.301087 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.301461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.301438 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-89k48\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.301637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.301617 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.301889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.301868 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.302097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.302076 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.302241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.302225 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.302431 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.302413 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.302570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.302554 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.302734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.302717 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.405870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.405745 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.405870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.405851 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-89k48\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.405904 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406013 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406063 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406093 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406170 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406206 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.406571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.406232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.411658 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.407216 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.411658 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.411573 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.411903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.411853 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.413657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.412082 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.413657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.412977 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.413657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.413441 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.423140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.423110 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-89k48\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.446818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.430976 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.466819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.451458 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.466819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.458451 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:16:03.485834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.476714 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.574836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.560665 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:03.632145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.631694 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.632145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.631744 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.635112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.632487 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.635112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.632566 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.635112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.632600 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzghm\" (UniqueName: \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.635112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.632637 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.643297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.643253 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w6-0402df\"/\"distinct-w6-0402df-kserve-self-signed-certs\"" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.737968 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.738019 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzghm\" (UniqueName: \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.738054 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.738297 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.738322 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.743915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.738371 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.751637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.751601 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.752220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.752131 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.774323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.774266 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:16:03.774745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.774725 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.873835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.853411 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.873835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.864745 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:03.889257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:03.888847 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzghm\" (UniqueName: \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\") pod \"distinct-w6-0402df-kserve-5c874c97f-nsk6p\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:04.175197 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:04.175167 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfed08b4a_3f3f_49e8_a91f_0e70ebffd914.slice/crio-ced68ad8c85358a0a19e4d33e0ceb7ad59786926a066920db22a09b8c8c0b181 WatchSource:0}: Error finding container ced68ad8c85358a0a19e4d33e0ceb7ad59786926a066920db22a09b8c8c0b181: Status 404 returned error can't find the container with id ced68ad8c85358a0a19e4d33e0ceb7ad59786926a066920db22a09b8c8c0b181 Oct 31 00:16:04.175622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.175572 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:16:04.178877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.178837 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:04.178958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.178939 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:04.178998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.178982 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:04.179926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.179906 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:04.514420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.514384 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:16:04.806931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.806831 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" event={"ID":"75668942-ad30-4251-ad4b-b5df786ff9ae","Type":"ContainerStarted","Data":"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085"} Oct 31 00:16:04.806931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.806893 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" event={"ID":"75668942-ad30-4251-ad4b-b5df786ff9ae","Type":"ContainerStarted","Data":"a90c624f075f64b037295dcb96b3af693300e6be8ae23aed7c7aba79da3ddf0d"} Oct 31 00:16:04.809493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.809459 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" event={"ID":"fed08b4a-3f3f-49e8-a91f-0e70ebffd914","Type":"ContainerStarted","Data":"67f3c104a63781011eeadebc8d2092605d535082e205077fba26e39a5e2f3d93"} Oct 31 00:16:04.809640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.809563 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" event={"ID":"fed08b4a-3f3f-49e8-a91f-0e70ebffd914","Type":"ContainerStarted","Data":"ced68ad8c85358a0a19e4d33e0ceb7ad59786926a066920db22a09b8c8c0b181"} Oct 31 00:16:04.828736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:04.828663 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podStartSLOduration=1.828647197 podStartE2EDuration="1.828647197s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:04.827734871 +0000 UTC m=+1286.860830941" watchObservedRunningTime="2026-10-31 00:16:04.828647197 +0000 UTC m=+1286.861743267" Oct 31 00:16:05.181159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:05.181069 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:05.577086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:05.577056 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:05.578515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:05.578486 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:05.578629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:05.578539 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:06.576383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:06.576352 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:06.576831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:06.576407 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:07.098533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.098482 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podStartSLOduration=4.098465035 podStartE2EDuration="4.098465035s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:04.858236078 +0000 UTC m=+1286.891332151" watchObservedRunningTime="2026-10-31 00:16:07.098465035 +0000 UTC m=+1289.131561105" Oct 31 00:16:07.101023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.101006 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:16:07.105001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.104984 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.113593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.113544 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"openshift-service-ca.crt\"" Oct 31 00:16:07.117017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.116999 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"kube-root-ca.crt\"" Oct 31 00:16:07.117281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.117262 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"default-dockercfg-hkvmb\"" Oct 31 00:16:07.117349 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.117338 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-4c360e\"/\"distinct-w1-4c360e-kserve-self-signed-certs\"" Oct 31 00:16:07.120680 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.120659 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:16:07.125585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125551 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.125697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125604 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.125760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125717 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xc7w6\" (UniqueName: \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.125843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125769 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.125906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125890 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.125963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.125917 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.229662 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xc7w6\" (UniqueName: \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.229740 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.229860 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.229891 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.230009 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.230046 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.232158 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.233031 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.255990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.233614 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.272160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.272127 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.281617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.281547 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.298835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.289090 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xc7w6\" (UniqueName: \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\") pod \"distinct-w1-4c360e-kserve-747d7954ff-s6d6b\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.419675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.419592 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:07.576827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.576365 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:07.576827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.576410 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:07.766944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.766915 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:16:07.831936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:07.831901 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" event={"ID":"78fb516f-09a1-4d73-a84d-6be7f78e360f","Type":"ContainerStarted","Data":"18d49fc0e10ce3b4fb06d494c60290b70963275c11a67a74a09e2223cbb4cded"} Oct 31 00:16:08.576209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:08.576177 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:08.576408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:08.576230 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:08.864772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:08.864668 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" event={"ID":"78fb516f-09a1-4d73-a84d-6be7f78e360f","Type":"ContainerStarted","Data":"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765"} Oct 31 00:16:08.885690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:08.885639 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podStartSLOduration=1.885624953 podStartE2EDuration="1.885624953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:08.885223328 +0000 UTC m=+1290.918319399" watchObservedRunningTime="2026-10-31 00:16:08.885624953 +0000 UTC m=+1290.918721068" Oct 31 00:16:09.420232 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:09.420203 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:09.576471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:09.576430 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:09.576672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:09.576534 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:10.576368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:10.576336 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body= Oct 31 00:16:10.576897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:10.576410 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" Oct 31 00:16:11.580974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:11.580937 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:11.581442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:11.581341 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:11.582068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:11.582049 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:16:14.186818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:14.181186 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:14.225271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:14.225227 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:14.912332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:14.912302 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:17.420204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.420166 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:17.433561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.433533 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:17.568135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.568105 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:16:17.572464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.572440 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.586682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.586655 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:16:17.603532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603499 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603552 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603616 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603637 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603680 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603847 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603922 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.603982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.603964 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.604072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.604055 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.704768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.704730 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.704991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.704850 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.704991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.704902 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.704991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.704934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.704991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.704981 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705102 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705146 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705215 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705325 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705394 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705811 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.705974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705955 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.706026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.705982 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.707998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.707975 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.708088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.708059 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.717985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.717963 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.718121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.718101 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.830707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.830670 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:16:17.836122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.836096 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.841238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.841213 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-d91de6-openshift-default-dockercfg-p87px\"" Oct 31 00:16:17.851616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.851586 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:16:17.896117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.896081 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.907542 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.907685 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.907752 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.907829 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.907904 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.908039 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.908081 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.908145 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.909462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.908173 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zkfxk\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:17.942452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:17.942421 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:16:18.010041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010002 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010076 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010160 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010194 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zkfxk\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010297 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010368 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010407 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010447 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010527 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010617 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.010911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.010884 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.011160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.011137 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.011235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.011191 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.011672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.011645 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.015150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.015120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.015631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.015578 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.023611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.023587 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.023691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.023632 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zkfxk\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.152396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.152355 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:18.186213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.186183 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:16:18.192665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.192623 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.192828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.192700 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.192828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.192730 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.441969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.441903 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:16:18.444688 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:18.444662 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1855f8df_d794_4f19_a790_5545eb7e1222.slice/crio-ae5a86cd209d9021726eb19ca1117114f55da11d7f9b7274b93e14770b67f471 WatchSource:0}: Error finding container ae5a86cd209d9021726eb19ca1117114f55da11d7f9b7274b93e14770b67f471: Status 404 returned error can't find the container with id ae5a86cd209d9021726eb19ca1117114f55da11d7f9b7274b93e14770b67f471 Oct 31 00:16:18.447330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.447296 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.447413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.447363 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.447413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.447394 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:18.946413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.946377 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" event={"ID":"1855f8df-d794-4f19-a790-5545eb7e1222","Type":"ContainerStarted","Data":"6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228"} Oct 31 00:16:18.946604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.946440 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" event={"ID":"1855f8df-d794-4f19-a790-5545eb7e1222","Type":"ContainerStarted","Data":"ae5a86cd209d9021726eb19ca1117114f55da11d7f9b7274b93e14770b67f471"} Oct 31 00:16:18.948770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.948741 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" event={"ID":"c46f2326-e7b7-4758-a19e-eaa02debd40e","Type":"ContainerStarted","Data":"c1ecfe013134fa8eab807008b4d3d85f866e7dc2c53609c274be2869f623465d"} Oct 31 00:16:18.948886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.948814 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" event={"ID":"c46f2326-e7b7-4758-a19e-eaa02debd40e","Type":"ContainerStarted","Data":"bf52e01be9d8e3f98526121075e20afd57d4ae9d97aaea4c406ebe094218d7ed"} Oct 31 00:16:18.972031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.971984 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podStartSLOduration=1.971971456 podStartE2EDuration="1.971971456s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:18.969515519 +0000 UTC m=+1301.002611601" watchObservedRunningTime="2026-10-31 00:16:18.971971456 +0000 UTC m=+1301.005067582" Oct 31 00:16:18.999678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:18.999629 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podStartSLOduration=1.999615522 podStartE2EDuration="1.999615522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:18.997346438 +0000 UTC m=+1301.030442505" watchObservedRunningTime="2026-10-31 00:16:18.999615522 +0000 UTC m=+1301.032711592" Oct 31 00:16:19.155810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.152708 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:19.157815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.156914 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:19.157815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.156981 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:19.898562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.897356 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:19.899089 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.899068 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:19.899213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:19.899180 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:20.157886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:20.157236 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:20.157886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:20.157301 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:20.911323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:20.910186 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:20.911323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:20.910265 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:21.160818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:21.160206 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:21.160818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:21.160273 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:21.896748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:21.896709 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:21.896942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:21.896775 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:22.154762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:22.154266 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:22.154762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:22.154338 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:22.899294 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:22.899252 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:22.899604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:22.899570 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:23.158561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:23.158469 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:23.159171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:23.159137 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:24.191636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.191596 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:24.192304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.192251 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:24.255952 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.255916 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:24.256276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.256228 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:24.569907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.561890 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:24.784809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.710758 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:24.786311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.775892 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:24.901411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.901303 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:24.901700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.901652 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:24.936638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.935909 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:24.937455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:24.937409 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:16:25.159106 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.159021 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:25.159370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.159336 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:25.226869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.226820 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:25.406222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.406108 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:16:25.411597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.411533 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.422350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.422323 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w2-7c5da3\"/\"distinct-2-w2-7c5da3-kserve-self-signed-certs\"" Oct 31 00:16:25.429912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.424336 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.430141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.430125 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2zhkp\" (UniqueName: \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.431663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.431637 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.431902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.431855 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.432001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.431968 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:16:25.432086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.432057 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.432150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.432108 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.548706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.548656 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2zhkp\" (UniqueName: \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.565812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.562234 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.567368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.566205 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.591937 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.591991 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.592233 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.592641 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.567829 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.593120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.581421 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2zhkp\" (UniqueName: \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.585893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.603603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.600174 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\") pod \"distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.751721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.751447 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:25.897381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.897346 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body= Oct 31 00:16:25.897541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:25.897404 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" Oct 31 00:16:26.141022 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:26.140988 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77222f3c_662b_4453_aa15_b2be294e2526.slice/crio-4d435af8b762c0e2f27dc871d54105c123e6db86e90636ce0c04e18c9c298aa7 WatchSource:0}: Error finding container 4d435af8b762c0e2f27dc871d54105c123e6db86e90636ce0c04e18c9c298aa7: Status 404 returned error can't find the container with id 4d435af8b762c0e2f27dc871d54105c123e6db86e90636ce0c04e18c9c298aa7 Oct 31 00:16:26.141165 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.141138 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:16:26.152736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.152700 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" start-of-body= Oct 31 00:16:26.152854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.152754 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": dial tcp 10.134.0.71:15021: connect: connection refused" Oct 31 00:16:26.247690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.247649 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" event={"ID":"77222f3c-662b-4453-aa15-b2be294e2526","Type":"ContainerStarted","Data":"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb"} Oct 31 00:16:26.248286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.248255 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" event={"ID":"77222f3c-662b-4453-aa15-b2be294e2526","Type":"ContainerStarted","Data":"4d435af8b762c0e2f27dc871d54105c123e6db86e90636ce0c04e18c9c298aa7"} Oct 31 00:16:26.272842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.272748 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podStartSLOduration=1.272729869 podStartE2EDuration="1.272729869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:26.271981391 +0000 UTC m=+1308.305077463" watchObservedRunningTime="2026-10-31 00:16:26.272729869 +0000 UTC m=+1308.305825940" Oct 31 00:16:26.753118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:26.753082 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:27.078553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.078475 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:27.078850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.078821 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:27.079677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.079657 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:16:27.158990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.158954 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:27.159695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.159551 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:27.160974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:27.160954 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:16:33.030195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.030155 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:16:33.035237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.035209 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.053778 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.053746 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:16:33.134203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134164 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134288 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134330 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134417 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5h8rt\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134515 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134561 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134700 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134746 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.134854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.134776 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.162239 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:16:33.162206 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248384 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248454 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248731 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248795 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248824 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.248918 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.249016 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.249044 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.249090 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5h8rt\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.250553 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.254984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.254553 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.257000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.256973 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.258614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.257893 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.262715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.262688 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.262990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.262969 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:16:33.272076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.272043 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.272537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.272517 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.273207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.273185 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.275060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.274642 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:16:33.275060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.274742 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.281910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.281828 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5h8rt\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.350682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350653 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.350904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350689 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.350904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350838 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.350904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350903 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8v54\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.351075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350930 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.351075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.350967 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.351149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.351110 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.351197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.351155 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.351197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.351174 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.353397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.353381 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:33.452233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.452191 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.452596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.452570 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.452758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.452741 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.452918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.452900 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453013 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.452648 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453109 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453217 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453365 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453460 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453734 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.453958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8v54\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.454030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.453988 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.454108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.454082 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.454716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.454688 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.457303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.457259 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.457441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.457419 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.466252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.466230 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.467243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.467220 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8v54\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.617533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.617460 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:33.641960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.641930 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:16:33.643542 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:33.643517 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd311da85_ff2b_4b37_aae3_2f268a63b866.slice/crio-a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7 WatchSource:0}: Error finding container a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7: Status 404 returned error can't find the container with id a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7 Oct 31 00:16:33.646536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.646503 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:33.646630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.646580 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:33.646630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.646623 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:33.912175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.912144 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:16:33.913379 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:33.913351 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf531d8ab_90a1_4433_8131_da5ac62a69eb.slice/crio-01f810395c6a0f3d7868eecfc7b9d78d001c82fc5656251e0935ad2fe915e01b WatchSource:0}: Error finding container 01f810395c6a0f3d7868eecfc7b9d78d001c82fc5656251e0935ad2fe915e01b: Status 404 returned error can't find the container with id 01f810395c6a0f3d7868eecfc7b9d78d001c82fc5656251e0935ad2fe915e01b Oct 31 00:16:33.916494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.916458 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:33.916569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.916533 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:33.916569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:33.916563 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:34.301497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.301460 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" event={"ID":"f531d8ab-90a1-4433-8131-da5ac62a69eb","Type":"ContainerStarted","Data":"6fd39e8a89ed5b6e7c76cb3a664ed7c5fb0fc9ae8d2302b4050f3a7a50600472"} Oct 31 00:16:34.302010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.301522 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" event={"ID":"f531d8ab-90a1-4433-8131-da5ac62a69eb","Type":"ContainerStarted","Data":"01f810395c6a0f3d7868eecfc7b9d78d001c82fc5656251e0935ad2fe915e01b"} Oct 31 00:16:34.303182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.303147 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" event={"ID":"d311da85-ff2b-4b37-aae3-2f268a63b866","Type":"ContainerStarted","Data":"c0f43c82964f9de5db2181d9315bd2ece5da5128fa2ca196db6df8b7986dbf6d"} Oct 31 00:16:34.303297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.303200 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" event={"ID":"d311da85-ff2b-4b37-aae3-2f268a63b866","Type":"ContainerStarted","Data":"a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7"} Oct 31 00:16:34.332852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.332798 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podStartSLOduration=1.332768864 podStartE2EDuration="1.332768864s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:34.330603544 +0000 UTC m=+1316.363699617" watchObservedRunningTime="2026-10-31 00:16:34.332768864 +0000 UTC m=+1316.365864933" Oct 31 00:16:34.355877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.355846 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:34.361808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.360734 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podStartSLOduration=2.360714476 podStartE2EDuration="2.360714476s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:34.357270753 +0000 UTC m=+1316.390366824" watchObservedRunningTime="2026-10-31 00:16:34.360714476 +0000 UTC m=+1316.393810547" Oct 31 00:16:34.370429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.370397 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:34.370671 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.370646 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:34.618080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.617999 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:34.619796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.619536 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:34.619796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:34.619580 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:35.361807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.361255 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:35.361807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.361322 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:35.618513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.618421 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:35.618513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.618481 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:35.760809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.757414 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:35.804759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:35.804675 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:36.327277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:36.327249 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:16:36.369809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:36.364549 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:36.369809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:36.364616 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:36.618691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:36.618614 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:36.618691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:36.618675 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:37.359811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:37.358220 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.359811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:37.358282 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:37.630859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:37.621177 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:37.630859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:37.621236 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:38.385809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:38.383481 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:38.385809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:38.383557 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:38.617958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:38.617920 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:38.618156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:38.617985 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:39.360810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:39.360179 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:39.360810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:39.360242 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:39.617978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:39.617875 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:39.617978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:39.617936 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:40.354213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:40.354176 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body= Oct 31 00:16:40.354386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:40.354231 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" Oct 31 00:16:40.618056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:40.617959 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body= Oct 31 00:16:40.618056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:40.618031 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" Oct 31 00:16:41.358080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.358042 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:41.358347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.358322 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:41.359183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.359163 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:16:41.622131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.622043 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:41.622566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.622411 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:41.623200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:41.623181 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:16:46.471810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.470949 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:46.566811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.566317 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:16:46.747875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.745887 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:46.758817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.758266 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:46.961809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.942981 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:46.974983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:46.974929 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:47.202937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.202889 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 00:16:47.316838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.315931 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:47.555103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.554992 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:16:47.604805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.602993 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:47.604805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.603543 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:16:47.611359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.610895 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:47.611359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.611201 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:16:47.709951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:47.709893 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:48.010265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.010224 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:16:48.011727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.011705 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:48.013775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.013758 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:16:48.320869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.293710 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:16:48.402001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.401965 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:16:48.410149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.410116 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.424442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.424396 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:16:48.424707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.424690 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:16:48.434008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.433979 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:16:48.434337 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.434320 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.686867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.661739 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:16:48.686867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.666877 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:16:48.686867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.685797 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:16:48.690898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.690868 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:16:48.691189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.691171 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.693601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.693579 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:16:48.698045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.698020 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.698378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.698355 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.698586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.698565 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.698773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.698751 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.698980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.698942 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.699151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.699132 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.699378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.699342 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.699556 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.699537 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.699727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.699709 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5pcr\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.699968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.699909 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.700151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.700133 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.700313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.700295 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.700493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.700476 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.700756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.700657 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.700756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.700712 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701076 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701111 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jrwvm\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701160 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmdkw\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701212 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701246 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701272 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701335 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701381 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701413 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701446 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701521 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.702976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.701548 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.785607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.785494 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802328 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802428 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802455 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802484 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802510 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802535 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802589 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802611 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802637 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5pcr\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802668 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802699 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802721 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802759 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802816 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802839 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802874 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802899 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jrwvm\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmdkw\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.802981 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803009 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803033 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803087 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803126 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803156 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803185 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803247 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.803396 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.804731 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.805378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.805844 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.806349 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.806447 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.806596 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.807180 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.807269 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.808500 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.808572 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.809560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.809094 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.812553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.809654 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.812553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.810811 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.812553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.811262 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.816413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.812994 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.816413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.815040 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.816988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.816746 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.816988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.816942 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.818492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.817608 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.824040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.824013 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.824675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.824651 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.829716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.828630 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jrwvm\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.829716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.829629 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\") pod \"e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:48.829716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.829695 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5pcr\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.830905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.830882 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:48.847889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.847851 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmdkw\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\") pod \"e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:48.887231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:48.887149 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:49.065512 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.065473 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:49.119218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.118926 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:49.190562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.190532 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:16:49.204121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.203927 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.204121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.203999 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.204121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.204028 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.361546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.361516 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:16:49.362033 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:49.362010 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod52c9f3ff_aaab_4e1b_9b8a_ff6c65fa37e0.slice/crio-2d4c40a85adebf5e047e663125b8ceb12f0441d8614856cb8ac4b620b687a202 WatchSource:0}: Error finding container 2d4c40a85adebf5e047e663125b8ceb12f0441d8614856cb8ac4b620b687a202: Status 404 returned error can't find the container with id 2d4c40a85adebf5e047e663125b8ceb12f0441d8614856cb8ac4b620b687a202 Oct 31 00:16:49.364544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.364505 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.364655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.364584 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.364655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.364628 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.436587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.436558 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:16:49.437270 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:16:49.437234 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c4382fe_0417_4b17_be3f_b4bd257b8876.slice/crio-278a71a5e2c6b461fc75ee74c3804577daee5d5d5075a551b195b9b306b8bfb4 WatchSource:0}: Error finding container 278a71a5e2c6b461fc75ee74c3804577daee5d5d5075a551b195b9b306b8bfb4: Status 404 returned error can't find the container with id 278a71a5e2c6b461fc75ee74c3804577daee5d5d5075a551b195b9b306b8bfb4 Oct 31 00:16:49.444709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.444665 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.444870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.444761 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.444870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.444854 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:16:49.998287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.998242 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" event={"ID":"8c4382fe-0417-4b17-be3f-b4bd257b8876","Type":"ContainerStarted","Data":"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7"} Oct 31 00:16:49.998904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:49.998865 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" event={"ID":"8c4382fe-0417-4b17-be3f-b4bd257b8876","Type":"ContainerStarted","Data":"278a71a5e2c6b461fc75ee74c3804577daee5d5d5075a551b195b9b306b8bfb4"} Oct 31 00:16:50.002973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.002940 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" event={"ID":"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0","Type":"ContainerStarted","Data":"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013"} Oct 31 00:16:50.003164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.003150 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" event={"ID":"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0","Type":"ContainerStarted","Data":"2d4c40a85adebf5e047e663125b8ceb12f0441d8614856cb8ac4b620b687a202"} Oct 31 00:16:50.008813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.006759 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" event={"ID":"108ff184-1289-4ff7-9fb2-f7de9e5bf905","Type":"ContainerStarted","Data":"261d05543263a4e2fecebe04a46771ba41d2544e60b1e74a42779d912524366f"} Oct 31 00:16:50.009035 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.009011 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" event={"ID":"108ff184-1289-4ff7-9fb2-f7de9e5bf905","Type":"ContainerStarted","Data":"764f5becef65109eff22a50db07dfd86861f679fd7f0ce7e1af1510bd43d4c5a"} Oct 31 00:16:50.111815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.074524 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podStartSLOduration=3.074501802 podStartE2EDuration="3.074501802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:50.039997607 +0000 UTC m=+1332.073093689" watchObservedRunningTime="2026-10-31 00:16:50.074501802 +0000 UTC m=+1332.107597872" Oct 31 00:16:50.111815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.084132 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:50.111815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.086116 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:50.111815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.086164 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:50.134713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.134675 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:50.135007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.134977 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:50.213759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.207707 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:50.213759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.207798 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:16:50.213759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.211226 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podStartSLOduration=3.21120642 podStartE2EDuration="3.21120642s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:50.156650575 +0000 UTC m=+1332.189746646" watchObservedRunningTime="2026-10-31 00:16:50.21120642 +0000 UTC m=+1332.244302492" Oct 31 00:16:50.213759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.213084 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.220245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.220209 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:16:50.297878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.238910 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c60826\"/\"distinct-2-w0-c60826-kserve-self-signed-certs\"" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405133 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405251 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7zt7\" (UniqueName: \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405284 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405315 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405412 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.428186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.405464 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.488772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.486619 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podStartSLOduration=2.486596667 podStartE2EDuration="2.486596667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:50.283696306 +0000 UTC m=+1332.316792378" watchObservedRunningTime="2026-10-31 00:16:50.486596667 +0000 UTC m=+1332.519692738" Oct 31 00:16:50.521961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.521918 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.522278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.522245 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.522568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.522551 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.522756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.522740 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7zt7\" (UniqueName: \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.528980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.528957 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.529161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.529145 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.531241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.531217 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.533086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.531651 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.533226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.531886 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.538133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.538109 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.546808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.539113 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.578015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.555064 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7zt7\" (UniqueName: \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\") pod \"distinct-2-w0-c60826-kserve-b55d899-brb58\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.827763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.827726 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:50.890359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.890330 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:50.902808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.896905 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:50.902808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:50.896966 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:51.096864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.076217 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.096864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.076280 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:51.126834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.123200 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.126834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.123259 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:51.374861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.351603 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:16:51.896579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.893229 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:51.896579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:51.893288 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:52.069889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.066327 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.069889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.066403 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:52.122810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.121934 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.122810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.122020 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:52.142865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.139223 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" event={"ID":"c6d115f3-0494-4a5d-bf51-d3358c96c1b9","Type":"ContainerStarted","Data":"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644"} Oct 31 00:16:52.150915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.150877 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" event={"ID":"c6d115f3-0494-4a5d-bf51-d3358c96c1b9","Type":"ContainerStarted","Data":"4169e7aea03cbdd363dccd339fff83a886826b8dd2964150ef45e20cf77571e4"} Oct 31 00:16:52.213039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.212978 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podStartSLOduration=2.21295561 podStartE2EDuration="2.21295561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:16:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:16:52.162304649 +0000 UTC m=+1334.195400724" watchObservedRunningTime="2026-10-31 00:16:52.21295561 +0000 UTC m=+1334.246051681" Oct 31 00:16:52.828748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.828714 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:16:52.888012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.887538 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:52.888012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:52.887603 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:53.074811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.070269 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.074811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.070330 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:53.121886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.120255 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.121886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.120320 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:53.896203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.896166 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:53.896903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:53.896865 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:54.069252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.069216 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.069530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.069490 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:54.129806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.128071 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.129806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.128134 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:54.920407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.920368 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:54.921081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:54.921048 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:55.068810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.067186 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.068810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.067250 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:55.120494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.120464 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.120721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.120694 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:55.887981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.887925 6234 patch_prober.go:29] interesting pod/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" start-of-body= Oct 31 00:16:55.888189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:55.888012 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.77:15021/healthz/ready\": dial tcp 10.134.0.77:15021: connect: connection refused" Oct 31 00:16:56.069840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:56.068209 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" start-of-body= Oct 31 00:16:56.069840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:56.068268 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": dial tcp 10.134.0.75:15021: connect: connection refused" Oct 31 00:16:56.120859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:56.120825 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" start-of-body= Oct 31 00:16:56.121012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:56.120875 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": dial tcp 10.134.0.76:15021: connect: connection refused" Oct 31 00:16:57.157235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.156042 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:57.157235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.157003 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:57.160466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.160411 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:16:57.368916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.368880 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:57.369828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.369770 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:57.370979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.370955 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:16:57.487725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.487700 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:57.489246 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.489182 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:16:57.492536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:16:57.492512 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:17:00.828328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:00.828291 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:00.841900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:00.841874 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:01.348158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:01.348129 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:03.222030 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:17:03.221972 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:17:11.042255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.040948 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:17:11.142067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.140865 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:11.325993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.325885 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.341149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.341082 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:11.429939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.429887 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.706944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.706899 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.709689 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.707194 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.750945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.750900 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.840253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.840189 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:11.939374 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.901380 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 31 00:17:11.940766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:11.940589 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.065902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.045891 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.065902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.046297 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:12.322904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.300318 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:12.366235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.366188 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.421620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.421567 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.421948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.392041 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.422172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.422138 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.460808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.440227 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.485995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.485944 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.769809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.764912 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.770194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.770162 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:12.770404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.770376 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.793158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.793108 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:12.794465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.794429 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.862809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.825261 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:12.885085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.885051 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:12.888804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.886421 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:12.888804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.886473 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:12.888804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.886506 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:12.888804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.886541 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:12.901914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.901866 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.902900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.902877 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:12.976124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.968942 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:12.976124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:12.969363 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:13.022972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.022886 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:13.035737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.035708 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:13.145806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.113430 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:13.255819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.247260 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:13.268883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.268855 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:13.330534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.330455 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:13.389177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.389149 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:13.389418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.389406 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:13.389623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:13.389612 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:17.794565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.794474 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:17:17.801230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.801207 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.817346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.817316 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:17:17.829650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.829624 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:17:17.838427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.838400 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:17.854134 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.854105 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:17:17.912906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.912872 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.912933 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913026 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913082 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6fgvm\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913226 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913384 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913410 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913447 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.913537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.913498 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:17.964747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.964715 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:17:17.969122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.969096 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:17.990461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:17.990433 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:17:18.015208 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015168 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.015394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015292 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.015394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015337 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.015503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015404 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.015503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015479 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6fgvm\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.015712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015528 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.015712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015686 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.015923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015749 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.015923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.015810 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.016174 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016112 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.016238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016180 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.016288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016267 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.016339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016302 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.016389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016347 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.016441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016415 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.016490 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016468 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.016547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016508 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.016653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016622 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.016985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.016949 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.017143 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.017100 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.017402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.017378 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxwnn\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.017701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.017672 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.019538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.019504 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.023055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.023022 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.023865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.023838 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.031770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.031742 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.033056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.033013 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6fgvm\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.042015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.041976 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:17:18.051674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.051608 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.053312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.053279 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:17:18.116794 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.116744 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:18.119468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119435 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.119635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119482 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.119635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119550 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.119755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119704 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.119755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119747 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.119885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119773 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119882 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.119965 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120005 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120033 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120050 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c8nhk\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120099 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120111 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120132 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120145 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120182 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120257 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120313 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120472 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxwnn\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.120572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120535 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.121402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120877 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.121402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.120952 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.121402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.121113 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.122540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.122516 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.122962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.122941 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.135050 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.135021 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.135679 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.135602 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxwnn\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\") pod \"e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.158326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.158292 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.222824 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.222899 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.222932 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.222982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223032 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f6psh\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223103 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223165 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223194 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c8nhk\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223242 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223315 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223350 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223393 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223444 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223492 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223533 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223612 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223652 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.224504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.223750 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.225889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.225804 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.225889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.225869 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.226317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.226246 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.226754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.226544 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.226754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.226716 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.226972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.226913 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.228265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.228223 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.247906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.247880 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.248173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.248144 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c8nhk\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.289875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.289831 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325368 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325405 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325464 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f6psh\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325544 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325665 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325707 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325828 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.325877 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.326028 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.326443 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.326653 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.326952 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.327171 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.327408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.327286 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.329609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.329560 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.330424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.330402 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:17:18.331572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.331513 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.348695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.348174 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.350611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.350582 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:17:18.360265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.360204 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f6psh\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.376041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.375902 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.425635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.425608 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:17:18.429891 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:17:18.429860 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb5a7af4_a5e7_444a_a4f6_bc2d60bb8653.slice/crio-a69d0540ce8580745ea11cf8f050de2a3dc8a643a3e4bb81805e1d6707a8f4f8 WatchSource:0}: Error finding container a69d0540ce8580745ea11cf8f050de2a3dc8a643a3e4bb81805e1d6707a8f4f8: Status 404 returned error can't find the container with id a69d0540ce8580745ea11cf8f050de2a3dc8a643a3e4bb81805e1d6707a8f4f8 Oct 31 00:17:18.432534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.432496 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.432637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.432578 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.432637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.432629 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.486821 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:17:18.485898 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod03c4be57_3cdc_49d9_8612_069310252339.slice/crio-d0286bf7cfb7ad60310f2cfdf003e2992741a63127c49f7910410bedc441de4f WatchSource:0}: Error finding container d0286bf7cfb7ad60310f2cfdf003e2992741a63127c49f7910410bedc441de4f: Status 404 returned error can't find the container with id d0286bf7cfb7ad60310f2cfdf003e2992741a63127c49f7910410bedc441de4f Oct 31 00:17:18.487840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.487691 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:17:18.488970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.488942 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.489032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.489006 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.489065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.489035 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.529599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.529562 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.529748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.529677 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.529880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.529828 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.529974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.529959 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.530039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.529991 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.530120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.530096 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.530226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.530201 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.530314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.530296 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lk55t\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.530432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.530417 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.614326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.614293 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:17:18.616497 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:17:18.616464 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8e2ec100_4d8c_4cce_974c_18ab8b60f3cc.slice/crio-d6966a92ac743243492d3cb311e967b8c0d5a718fb85eaa1c477463d715ff1b9 WatchSource:0}: Error finding container d6966a92ac743243492d3cb311e967b8c0d5a718fb85eaa1c477463d715ff1b9: Status 404 returned error can't find the container with id d6966a92ac743243492d3cb311e967b8c0d5a718fb85eaa1c477463d715ff1b9 Oct 31 00:17:18.619590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.619546 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.619711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.619636 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.619711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.619674 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631501 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lk55t\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631575 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631638 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631713 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631762 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631877 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631917 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.631960 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.632018 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.632567 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.632752 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.632842 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.633218 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.633508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.633470 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.636749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.636724 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.639395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.639371 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.646173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.646108 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lk55t\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.648234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.648208 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\") pod \"e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.678025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.677996 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:18.702738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.702702 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:18.904361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.904322 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" event={"ID":"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653","Type":"ContainerStarted","Data":"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6"} Oct 31 00:17:18.907764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.907253 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" event={"ID":"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653","Type":"ContainerStarted","Data":"a69d0540ce8580745ea11cf8f050de2a3dc8a643a3e4bb81805e1d6707a8f4f8"} Oct 31 00:17:18.911488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.911347 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" event={"ID":"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc","Type":"ContainerStarted","Data":"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba"} Oct 31 00:17:18.911488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.911401 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" event={"ID":"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc","Type":"ContainerStarted","Data":"d6966a92ac743243492d3cb311e967b8c0d5a718fb85eaa1c477463d715ff1b9"} Oct 31 00:17:18.913977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.913951 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" event={"ID":"03c4be57-3cdc-49d9-8612-069310252339","Type":"ContainerStarted","Data":"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a"} Oct 31 00:17:18.914064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.914000 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" event={"ID":"03c4be57-3cdc-49d9-8612-069310252339","Type":"ContainerStarted","Data":"d0286bf7cfb7ad60310f2cfdf003e2992741a63127c49f7910410bedc441de4f"} Oct 31 00:17:18.944068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:18.944014 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podStartSLOduration=1.943998722 podStartE2EDuration="1.943998722s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:18.940298872 +0000 UTC m=+1360.973394943" watchObservedRunningTime="2026-10-31 00:17:18.943998722 +0000 UTC m=+1360.977094792" Oct 31 00:17:19.117395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.117362 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:19.120945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.120916 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.121166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.121139 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:19.128675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.128573 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podStartSLOduration=2.128552711 podStartE2EDuration="2.128552711s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.125398167 +0000 UTC m=+1361.158494238" watchObservedRunningTime="2026-10-31 00:17:19.128552711 +0000 UTC m=+1361.161648781" Oct 31 00:17:19.130290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.130247 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podStartSLOduration=2.130233236 podStartE2EDuration="2.130233236s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:19.073568116 +0000 UTC m=+1361.106664187" watchObservedRunningTime="2026-10-31 00:17:19.130233236 +0000 UTC m=+1361.163329313" Oct 31 00:17:19.159428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.159394 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:19.161041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.161015 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.161156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.161094 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:19.293932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.292103 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:19.293932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.293493 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:19.293932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.293551 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:19.296276 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:17:19.296242 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a67dd4_224d_4701_84a3_48540c600bfe.slice/crio-b3dbf1b36b576ad03d1512f468ae6e091999be25cfb8fff2ebb237863afbd62e WatchSource:0}: Error finding container b3dbf1b36b576ad03d1512f468ae6e091999be25cfb8fff2ebb237863afbd62e: Status 404 returned error can't find the container with id b3dbf1b36b576ad03d1512f468ae6e091999be25cfb8fff2ebb237863afbd62e Oct 31 00:17:19.299999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.299964 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:19.300095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.300033 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:19.300095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.300082 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:19.301326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.301303 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:17:19.306677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.306649 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:17:19.319645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.319541 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:19.319882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.319824 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:19.320040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:19.320006 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:20.052897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.046844 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" event={"ID":"48a67dd4-224d-4701-84a3-48540c600bfe","Type":"ContainerStarted","Data":"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286"} Oct 31 00:17:20.052897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.047012 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" event={"ID":"48a67dd4-224d-4701-84a3-48540c600bfe","Type":"ContainerStarted","Data":"b3dbf1b36b576ad03d1512f468ae6e091999be25cfb8fff2ebb237863afbd62e"} Oct 31 00:17:20.260000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.186173 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.260000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.186243 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:20.260000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.186748 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.260000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.186808 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:20.260000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.229908 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podStartSLOduration=3.229886618 podStartE2EDuration="3.229886618s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:20.128671102 +0000 UTC m=+1362.161767195" watchObservedRunningTime="2026-10-31 00:17:20.229886618 +0000 UTC m=+1362.262982690" Oct 31 00:17:20.299808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.298928 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.299808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.299012 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:20.411526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.411434 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" event={"ID":"af16b26e-fdaa-491c-9827-22cfd36bf957","Type":"ContainerStarted","Data":"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c"} Oct 31 00:17:20.416884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.416848 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" event={"ID":"af16b26e-fdaa-491c-9827-22cfd36bf957","Type":"ContainerStarted","Data":"13041bc3726912cf8d5a260ee6e42fe1f946fa1c893a6553401f23affc4d9a3b"} Oct 31 00:17:20.713808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.713246 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:20.713808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.713316 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:20.725809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.722411 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.725809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.722470 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:20.725809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.722824 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:20.725809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:20.722862 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:21.118865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.118708 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.118865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.118798 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:21.195807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.193239 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:21.195807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.194484 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.195807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.194530 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:21.295911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.295867 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:21.332607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.297468 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.332882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.332847 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:21.333032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.308871 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:21.367915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.367865 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:21.430523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.430426 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:21.430912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.430750 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:17:21.431371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.430820 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:21.686102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.686009 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.693908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.693866 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:21.710969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.710934 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:21.711199 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.711172 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:21.784110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.784066 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:21.871220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:21.870885 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.122280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.122245 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.122920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.122890 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:22.176811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.175072 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.176811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.175131 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:22.193942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.193901 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.194533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.194507 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:22.196346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.196315 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.232772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.232711 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podStartSLOduration=4.232690814 podStartE2EDuration="4.232690814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:20.245023531 +0000 UTC m=+1362.278119614" watchObservedRunningTime="2026-10-31 00:17:22.232690814 +0000 UTC m=+1364.265786888" Oct 31 00:17:22.337056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.337014 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.339403 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.337705 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:22.362810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.341282 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.379526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.379442 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:22.380288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.370919 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.381063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.381045 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:22.418030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.398263 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.448109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.446872 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.448109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.447253 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:17:22.448109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.447578 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:22.448109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.447643 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:22.511807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.496920 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.511807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.496985 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:22.689368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.689337 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.689627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.689594 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:22.792840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.789501 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:22.792840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.789561 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:22.792840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.790467 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:22.792840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:22.791491 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:23.184867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.143897 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.184867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.152672 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.184867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.152726 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:23.239272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.239231 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.262033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.262001 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:23.268067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.261920 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:23.277567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.273903 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:23.277567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.275624 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:23.277567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.276452 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.367814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.361066 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.367814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.361136 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:23.402955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.402907 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.503127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.433957 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.503563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.434015 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:17:23.503944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.503918 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:23.509991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.475358 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.510547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.475399 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:23.797924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.796918 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.797924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.796978 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:23.799147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.798419 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:23.799378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.799356 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:23.799526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:23.799504 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:24.124500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.124398 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.145747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.124728 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:24.392917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.346509 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.392917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.346928 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:24.443693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.443647 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:24.529945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.445997 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.530291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.530254 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:24.530497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.475238 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:24.589821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.589744 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:17:24.661917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.505889 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:24.661917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.505912 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:17:24.738825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.475029 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:24.738825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.505819 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:24.738825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.619081 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:24.771737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.771692 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.772094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.772063 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:24.889938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.886569 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:24.889938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.886665 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:24.992066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:24.991994 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.278096 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.278002 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.292810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.285885 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:25.326998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.326958 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.327266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.327234 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:25.403810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.403221 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.481856 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:17:25.470045 6234 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.374s" Oct 31 00:17:25.538869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.537892 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.544809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.539381 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:25.574888 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.559173 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.574888 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.559281 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:25.643004 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.642955 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.711010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.710977 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.734918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.734866 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:25.736458 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736227 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:25.777846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.776247 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:25.777846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736314 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:17:25.805878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736336 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.805878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736283 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:25.805878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736373 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:17:25.825380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.736355 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:17:25.828001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:25.827138 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:26.019201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.019146 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:26.139139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.137833 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.139139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.137899 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:26.187815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.179830 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.187815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.181746 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:26.327809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.312700 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.327809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.312756 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:26.783890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.783259 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:26.817808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.816938 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:26.836858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.826039 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:26.861809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.861520 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:26.899506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.899467 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:26.899748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.899719 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:26.914823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.914761 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:26.945642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.936660 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.015826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:26.981873 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 00:17:27.015826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.003318 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:27.015826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.003563 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.015826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.006512 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.015826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.014340 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:27.023975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.023925 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.197810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.193524 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.197810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.193583 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:27.197810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.193668 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.197810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.193692 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:27.198303 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.198268 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 00:17:27.300810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.298924 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.300810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.298998 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:27.698806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.695246 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.698806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.695302 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:27.712814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.712743 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:27.712989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.712824 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:27.795808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.795133 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.827809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.826315 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:17:27.827809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.827179 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.884150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.884089 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.916036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.915897 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:27.944497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:27.944455 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.005101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.004987 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.014648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.014591 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.042379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.042327 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.124814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.119622 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.124814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.119678 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:28.163239 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.163205 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.163514 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.163471 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:28.185821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.181073 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:28.326010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.325915 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.332807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.326294 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:28.468911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.448391 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.686272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.686175 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.686527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.686484 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:28.713347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.713304 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:28.713522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.713379 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:28.804641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.804597 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.863870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.832191 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.964565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.964453 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:28.967635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.965803 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:28.978636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:28.970237 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:29.044526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.044145 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:29.050937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.050895 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:29.071244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.071186 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:17:29.126043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.126006 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.133817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.126479 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": dial tcp 10.134.0.79:15021: connect: connection refused" Oct 31 00:17:29.172378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.172219 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.172378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.172302 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" Oct 31 00:17:29.305880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.303518 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.305880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.303579 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": dial tcp 10.134.0.81:15021: connect: connection refused" Oct 31 00:17:29.442708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.440882 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:29.458083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.458051 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:29.705868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.700758 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.705868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.700879 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:29.728808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.707005 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:29.729077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.729042 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:29.731057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.731033 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:29.884810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:29.880954 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.026752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.024072 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.057809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.053883 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.071121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.071073 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.078074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.078034 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.118696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.118651 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:30.121769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.121743 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:30.332299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.332211 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:30.415501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.415469 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:30.445136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.445108 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:30.476819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.468477 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:30.501552 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.501523 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:17:30.521441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.521402 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:30.541817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.541500 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:30.543753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.542878 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:30.550815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.545971 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:30.575426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.574807 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:17:30.681992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.681668 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body= Oct 31 00:17:30.682246 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.682214 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" Oct 31 00:17:30.704112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.704078 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" start-of-body= Oct 31 00:17:30.704453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.704426 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": dial tcp 10.134.0.83:15021: connect: connection refused" Oct 31 00:17:30.729093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.729061 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:30.733147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.733096 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:30.736437 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:30.736413 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:17:31.683129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.683099 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:31.683537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.683413 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:31.684191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.684170 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:17:31.708709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.708677 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:31.709040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.709023 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:31.709818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:31.709770 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:17:33.163373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.163327 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:17:33.166998 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:17:33.166963 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:17:33.168327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.168307 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.187299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.186254 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:17:33.214305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214264 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214316 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214370 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214419 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214511 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214699 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214808 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-68kmt\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214846 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.214957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.214914 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.316629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316586 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.316827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316689 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-68kmt\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.316827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316719 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.316827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316750 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316901 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.316940 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.317024 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.317084 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.317243 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.317316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.317462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.317440 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.318186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.318156 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.318461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.318199 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.318461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.318317 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.320196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.320174 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.320196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.320189 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.334391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.334362 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.334391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.334378 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-68kmt\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.496520 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.496488 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:33.796777 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.796747 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:17:33.800700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.800665 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:33.800844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.800733 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:33.800844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:33.800765 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:34.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.291803 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" event={"ID":"254ed5a9-9a0f-4918-b81e-6f71adcb8dea","Type":"ContainerStarted","Data":"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8"} Oct 31 00:17:34.292434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.292407 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" event={"ID":"254ed5a9-9a0f-4918-b81e-6f71adcb8dea","Type":"ContainerStarted","Data":"61ae280cd3ce1d8fa494d23db6809e7cc3a31a56f576a76a800cdc31783e1316"} Oct 31 00:17:34.328248 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.328193 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podStartSLOduration=1.328174765 podStartE2EDuration="1.328174765s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:34.325223968 +0000 UTC m=+1376.358320046" watchObservedRunningTime="2026-10-31 00:17:34.328174765 +0000 UTC m=+1376.361270841" Oct 31 00:17:34.496857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.496824 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:34.498246 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.498226 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:34.498326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:34.498274 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:35.524836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:35.518738 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:35.524836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:35.518823 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:36.497414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:36.497371 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:36.497579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:36.497429 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:37.497318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:37.497276 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:37.497832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:37.497341 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:38.497454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:38.497421 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:38.497900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:38.497472 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:39.497725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:39.497693 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:39.498164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:39.497746 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:40.497292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:40.497254 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" start-of-body= Oct 31 00:17:40.497441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:40.497326 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": dial tcp 10.134.0.84:15021: connect: connection refused" Oct 31 00:17:41.501971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:41.501942 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:41.502435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:41.502351 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:41.503123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:41.503106 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:17:44.006808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.005900 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.020165 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.020121 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:44.528978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.526868 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:17:44.528978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.527278 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:44.528978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.528695 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:44.546764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.546376 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:44.629811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.628935 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:17:44.664921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.664873 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:17:44.784170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:44.782884 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:17:45.006939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.006886 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.008276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.008250 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:45.033986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.033945 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.034686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.034618 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:45.210028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.208130 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.576811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.533272 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:45.651810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.544856 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.652347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.544893 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:17:45.653344 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.588711 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:17:45.653775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.628903 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:45.659088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.653991 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:45.685744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.685717 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:45.716810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.715932 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:45.717009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.670873 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:17:45.721027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.629537 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.721027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.718201 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:45.721027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.718266 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:45.793627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.786898 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:45.793627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:45.791919 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:46.018845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.016192 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.229809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.222017 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:17:46.540640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.540594 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.540955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.540921 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.656946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.656897 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.671929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.671886 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:46.687562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.672066 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:17:46.687562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.672103 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.719764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.719723 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:17:46.747848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.720166 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:17:46.813873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.813493 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 00:17:46.838816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.815544 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:17:46.869815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.849203 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.869815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.849287 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.79:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:46.869815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.856858 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:17:46.869815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:46.856918 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.023818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.019331 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.231858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.230753 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:17:47.248843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.232128 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.472811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.443146 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:17:47.696911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.689001 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.704481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.704440 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:17:47.704730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.704698 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.705202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.705179 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:17:47.740826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.731945 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.740826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.732724 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.740826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.737153 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:17:47.770323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.766741 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:17:47.780990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.778503 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:17:47.785329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.785220 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:17:47.812334 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.812272 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:17:47.824428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.824401 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:17:47.830166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.830140 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:17:47.861616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.861572 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:17:47.884330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:47.884298 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:17:48.148255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.148182 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:17:48.153722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.153698 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.170234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.170171 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:17:48.210402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210367 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zswk9\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210441 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210565 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210612 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210806 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210880 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.210947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.210930 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.211054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.211037 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312093 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zswk9\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312166 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312197 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312219 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312238 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312290 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312331 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312393 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312479 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312658 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312677 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312759 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.312925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.312884 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.313184 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.313161 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.314913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.314890 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.315254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.315225 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.325709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.325682 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.326068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.326045 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zswk9\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\") pod \"e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.471851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.471811 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:48.763997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.763949 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:17:48.766023 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:17:48.765994 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05affd62_3cea_4900_a6e0_cb30d73b1948.slice/crio-0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d WatchSource:0}: Error finding container 0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d: Status 404 returned error can't find the container with id 0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d Oct 31 00:17:48.768884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.768846 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:48.768989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.768926 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:48.768989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.768956 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:17:48.959423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.959390 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" event={"ID":"05affd62-3cea-4900-a6e0-cb30d73b1948","Type":"ContainerStarted","Data":"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3"} Oct 31 00:17:48.959616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.959444 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" event={"ID":"05affd62-3cea-4900-a6e0-cb30d73b1948","Type":"ContainerStarted","Data":"0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d"} Oct 31 00:17:48.989596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:48.989539 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podStartSLOduration=0.989519093 podStartE2EDuration="989.519093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:17:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:17:48.985762274 +0000 UTC m=+1391.018858367" watchObservedRunningTime="2026-10-31 00:17:48.989519093 +0000 UTC m=+1391.022615165" Oct 31 00:17:49.472313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:49.472280 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:49.473446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:49.473419 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:49.473572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:49.473471 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:50.472592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:50.472552 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:50.473061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:50.472620 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:51.473090 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:51.473050 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:51.473523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:51.473111 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:52.472289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:52.472244 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:52.472497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:52.472314 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:53.473119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:53.473083 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:53.473536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:53.473133 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:54.473200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:54.473163 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:54.473710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:54.473228 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:55.482811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:55.477982 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" start-of-body= Oct 31 00:17:55.482811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:55.478043 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": dial tcp 10.134.0.85:15021: connect: connection refused" Oct 31 00:17:56.477847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:56.477813 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:56.478300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:56.478279 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:17:56.478975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:17:56.478957 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:18:02.789816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.788849 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:18:02.820038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.820003 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:18:02.820378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.820360 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.825591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.825560 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:18:02.886325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.886298 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:02.925353 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.925307 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:18:02.937322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.937291 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.937540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.937523 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.937692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.937680 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.937823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.937808 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.938281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.938259 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.938478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.938465 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.938597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.938586 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.938744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.938733 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:02.938886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:02.938874 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fs4jc\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041061 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041179 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041250 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041301 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.041559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041356 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.041559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041402 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041532 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fs4jc\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041572 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.041719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041662 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041720 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.041845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041734 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.041845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041760 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.041845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041764 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.042057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041928 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.042057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.041975 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg678\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.042057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.042057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042031 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.042209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042166 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.042292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042263 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.042370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042268 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.042370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042320 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.042624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.042606 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.043930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.043909 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.044659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.044635 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.066953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.066923 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.067097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.066976 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fs4jc\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.143592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.143546 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.143759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.143713 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.143844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.143820 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.143894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.143882 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.143947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.143923 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144034 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg678\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144073 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144103 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144187 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144418 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144592 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144597 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144843 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.144961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.144890 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.146895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.146876 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.147054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.147036 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.158663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.158636 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.159956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.159934 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg678\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\") pod \"e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.166513 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:18:03.166478 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:18:03.233204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.233168 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:03.249077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.249047 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:03.530978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.530949 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:18:03.532165 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:18:03.532140 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod506e0f41_df4b_43ab_8e7c_7e3c33378153.slice/crio-6d5465a30a12d02b633e2b321cc5ef4610ba5275ef77b53907ef17862fd89d2a WatchSource:0}: Error finding container 6d5465a30a12d02b633e2b321cc5ef4610ba5275ef77b53907ef17862fd89d2a: Status 404 returned error can't find the container with id 6d5465a30a12d02b633e2b321cc5ef4610ba5275ef77b53907ef17862fd89d2a Oct 31 00:18:03.534966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.534924 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:03.535046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.534991 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:03.535046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.535028 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:03.552055 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:18:03.552022 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58ea5482_2d1b_4bd7_9f50_103331af8349.slice/crio-fcc0bf27887f096fd7d72d471e0409a3837708731e4aca0e9a848fdd1d090c66 WatchSource:0}: Error finding container fcc0bf27887f096fd7d72d471e0409a3837708731e4aca0e9a848fdd1d090c66: Status 404 returned error can't find the container with id fcc0bf27887f096fd7d72d471e0409a3837708731e4aca0e9a848fdd1d090c66 Oct 31 00:18:03.552292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.552268 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:18:03.554997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.554955 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:03.555083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.555047 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:03.555117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:03.555096 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:04.085708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.085663 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" event={"ID":"58ea5482-2d1b-4bd7-9f50-103331af8349","Type":"ContainerStarted","Data":"8b1836f2b728038dfffda3e1bbbd9474f3a3fbc91d7390cae532736cb42ca0b9"} Oct 31 00:18:04.086196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.085727 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" event={"ID":"58ea5482-2d1b-4bd7-9f50-103331af8349","Type":"ContainerStarted","Data":"fcc0bf27887f096fd7d72d471e0409a3837708731e4aca0e9a848fdd1d090c66"} Oct 31 00:18:04.086196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.086004 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" event={"ID":"506e0f41-df4b-43ab-8e7c-7e3c33378153","Type":"ContainerStarted","Data":"b1b4a3e33b67cb4722cb973e1bb866b05387c2d29ce849eb3b6095b8b96d9c6f"} Oct 31 00:18:04.086196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.086064 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" event={"ID":"506e0f41-df4b-43ab-8e7c-7e3c33378153","Type":"ContainerStarted","Data":"6d5465a30a12d02b633e2b321cc5ef4610ba5275ef77b53907ef17862fd89d2a"} Oct 31 00:18:04.113092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.113031 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podStartSLOduration=2.113012865 podStartE2EDuration="2.113012865s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.109863164 +0000 UTC m=+1406.142959215" watchObservedRunningTime="2026-10-31 00:18:04.113012865 +0000 UTC m=+1406.146108940" Oct 31 00:18:04.142587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.142521 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podStartSLOduration=2.142502394 podStartE2EDuration="2.142502394s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:04.139763091 +0000 UTC m=+1406.172859162" watchObservedRunningTime="2026-10-31 00:18:04.142502394 +0000 UTC m=+1406.175598466" Oct 31 00:18:04.234179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.234141 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:04.235713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.235689 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:04.235841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.235739 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:04.252142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.252113 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:04.256959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.256917 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:04.257224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:04.257195 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:05.254877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:05.252698 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:05.254877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:05.252770 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:05.267816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:05.256732 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:05.267816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:05.256815 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:06.233865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:06.233824 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:06.234052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:06.233889 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:06.261818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:06.259890 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:06.261818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:06.259955 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:07.234026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:07.233993 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:07.234181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:07.234042 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:07.250102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:07.250063 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:07.250383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:07.250352 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:08.233891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:08.233848 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:08.234295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:08.233914 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:08.250362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:08.250235 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:08.250362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:08.250295 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:09.234026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:09.233989 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:09.234444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:09.234048 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:09.253378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:09.253342 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:09.253616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:09.253584 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:10.233985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:10.233946 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body= Oct 31 00:18:10.234170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:10.234004 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" Oct 31 00:18:10.250333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:10.250300 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body= Oct 31 00:18:10.250461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:10.250361 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" Oct 31 00:18:11.368251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.367421 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:11.370885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.370861 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:11.374234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.374209 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:18:11.403113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.403067 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:11.404605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.404571 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:11.414161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:11.414132 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:18:15.663717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:15.663671 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:15.703432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:15.703388 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:15.703978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:15.703948 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:15.727975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:15.727907 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.030962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.030912 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.717635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.717583 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.888560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.725913 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:16.890666 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.890643 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:18:16.890964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.890947 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.856615 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.727667 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.856733 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.857813 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.888323 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.856712 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:18:16.894818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.856688 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:16.945829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.945317 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:18:16.945829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:16.945389 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:18:17.061911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.036898 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:17.061911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.048577 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:18:17.426091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.425967 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.906688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.906643 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:17.907347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.907310 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.920953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.920904 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:18:17.921575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.921523 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:17.922087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.922047 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.922554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.922532 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:17.922942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.922918 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:18:17.923364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.923341 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:17.923774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.923749 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:17.929812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.924181 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:18:17.930400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.930380 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:18:17.931619 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.931600 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:18:17.953977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.953930 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:17.954628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:17.954602 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:18:18.242124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:18.242082 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:18.575859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:18.537233 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 31 00:18:18.966957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:18.955470 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.044309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.044265 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.046230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.046197 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.046852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.046827 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.047036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.047015 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.047590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.047566 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:19.047759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.047741 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:18:19.047947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.047928 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:19.149815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.097627 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:18:19.177319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.177273 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:18:19.296436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.296202 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.298376 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.298459 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.299059 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.301320 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.301459 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306645 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306692 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh4fb\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306737 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306902 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306934 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.306990 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.307039 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.307192 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.327595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.307219 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.349166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.348585 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:18:19.363395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.363369 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:18:19.389669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.389388 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:18:19.398072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.398047 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:18:19.408670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.408636 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.408882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.408683 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.409339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.409315 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.409415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.409366 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh4fb\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.409480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.409421 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.409591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.409563 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410061 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410107 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410201 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410255 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.410926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.410851 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.411082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.411063 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.411433 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.411416 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.412163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.412125 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.413498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.413474 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.433563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.433527 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh4fb\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.438169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.438140 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\") pod \"e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:19.699910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:19.699880 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:20.038823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:20.038766 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:18:20.039239 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:18:20.039214 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6a93d93_30dc_43fc_a3a3_b640229dfc1f.slice/crio-3a2844622ea69b226fe604f87675b9ae8fb6ea05909a65dd0430b9ea906e2d88 WatchSource:0}: Error finding container 3a2844622ea69b226fe604f87675b9ae8fb6ea05909a65dd0430b9ea906e2d88: Status 404 returned error can't find the container with id 3a2844622ea69b226fe604f87675b9ae8fb6ea05909a65dd0430b9ea906e2d88 Oct 31 00:18:20.041869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:20.041824 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:20.041947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:20.041904 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:20.041947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:20.041935 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:18:20.053016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:20.052983 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" event={"ID":"d6a93d93-30dc-43fc-a3a3-b640229dfc1f","Type":"ContainerStarted","Data":"3a2844622ea69b226fe604f87675b9ae8fb6ea05909a65dd0430b9ea906e2d88"} Oct 31 00:18:21.071386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:21.071344 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" event={"ID":"d6a93d93-30dc-43fc-a3a3-b640229dfc1f","Type":"ContainerStarted","Data":"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e"} Oct 31 00:18:21.700314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:21.700284 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:21.701393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:21.701362 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:21.701516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:21.701410 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:22.700287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:22.700251 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:22.700767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:22.700321 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:23.714137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:23.714097 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:23.714702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:23.714668 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:24.703227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:24.703189 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:24.703609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:24.703578 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:25.704534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:25.704081 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:25.704534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:25.704151 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:26.700892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:26.700853 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body= Oct 31 00:18:26.701102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:26.700920 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" Oct 31 00:18:27.704696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:27.704664 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:27.705544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:27.705521 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:27.706278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:27.706257 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:18:27.742164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:27.742097 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podStartSLOduration=9.74208206 podStartE2EDuration="9.74208206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:18:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:18:21.101222923 +0000 UTC m=+1423.134318994" watchObservedRunningTime="2026-10-31 00:18:27.74208206 +0000 UTC m=+1429.775178130" Oct 31 00:18:33.162961 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:18:33.162921 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:18:35.969943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:35.969892 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:35.980945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:35.980897 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.076843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.065451 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:36.376074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.375935 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.398004 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.397940 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.770227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.769429 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.771091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.771044 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.942945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.941894 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.942945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.941966 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.982000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.979931 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:36.982000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.980399 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:18:37.004947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:36.993861 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:37.031898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.005566 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:18:37.099809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.095912 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:37.099809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.099145 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:18:37.235176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.234950 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": context deadline exceeded" Oct 31 00:18:37.307175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.307008 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.405912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.404993 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.405912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.405570 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:18:37.429116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.406887 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:37.429116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.407506 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:18:37.996810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:37.992114 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:38.077809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.072933 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:18:38.332561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.332448 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:38.393432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.376190 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:38.474228 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.474181 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp" podUID="8d2df4b9-ac3f-474f-a0f6-788c22573929" containerName="main" probeResult="failure" output="Get \"https://10.134.0.38:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:38.502365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.502320 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:38.502720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.502699 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:18:38.532047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.532012 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:18:38.604572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.600174 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:18:38.693853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.680961 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:38.970905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.950176 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" podUID="354612ef-bb88-4c36-b5fe-caf0ca868ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 00:18:38.970905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.784887 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:38.970905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.819069 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:38.970905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.858389 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-simulated-kserve-7b998b79d9-7mjdd" podUID="76eccd75-30b9-4a48-88ec-d7aca906afa4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.40:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.019905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:38.725633 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.019905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.001987 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:18:39.019905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.002049 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:18:39.019905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.007693 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.084810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.084202 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:39.869120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.869076 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:18:40.009813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.927179 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:40.014320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:39.961908 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:40.015276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.012896 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.145877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.137665 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.773814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.772513 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:40.773814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.773380 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 31 00:18:40.773814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.773655 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:40.774348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.774305 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:40.774611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.774584 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:40.868879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.868835 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:40.893809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:40.870277 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.092029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.091915 6234 patch_prober.go:29] interesting pod/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:41.098016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.097962 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.61:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.197732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.197680 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.213486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.213444 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:41.222103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.222051 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:41.247972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.222492 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:18:41.774250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.769095 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.796983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.796936 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:41.805808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.802269 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:41.818309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.818264 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:41.818878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.818850 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:41.852813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.823686 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:41.852813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.829352 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.852813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.846166 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:41.957829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:41.934530 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.210873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.209603 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.238169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.238124 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:42.240027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.239993 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.271833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.271760 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.314810 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:18:42.311680 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05affd62_3cea_4900_a6e0_cb30d73b1948.slice/crio-0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d\": RecentStats: unable to find data in memory cache]" Oct 31 00:18:42.498959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.472300 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.498959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.472366 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.498959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.473535 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.498959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.473585 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.516333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.516296 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.516595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.516565 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.552810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.540641 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.552810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.540701 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.85:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.583997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.582320 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.583997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.582393 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.84:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.712505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.711297 6234 patch_prober.go:29] interesting pod/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.712505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.711356 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.771366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.744264 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:42.771366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.744323 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.824811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.818268 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.850811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.825424 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.852512 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.825906 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.853881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.840400 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:42.938605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:42.938560 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.179878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.179752 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.226302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.226256 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:18:43.227281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.227249 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:43.227488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.227461 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.231920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.231894 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:43.271283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.271236 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.271547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.260937 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:43.272050 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.260896 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.272502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.272471 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.766879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.762911 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:18:43.766879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.762975 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:43.956812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.933322 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:43.960810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.958549 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:43.960810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.960492 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:18:43.961709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.961434 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:43.974431 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:43.974396 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:18:44.348021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:44.282558 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:44.348021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:44.307230 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:18:44.348021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:44.307604 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:18:44.348021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:44.314677 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:18:45.012810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.012249 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:18:45.058636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.058596 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:18:45.059185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.059152 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:45.060647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.060618 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:18:45.117760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.116198 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:18:45.118571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.118483 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:18:45.162808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.141144 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:18:45.180309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.180278 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:18:45.194486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.194441 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:18:45.241613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.241578 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:18:45.244702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.244584 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:18:45.249942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.249919 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:18:45.254279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.254259 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:18:45.257213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:18:45.257195 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:19:02.099936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.099843 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:02.104284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.104263 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.127746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.127714 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2d8f0cae\"" Oct 31 00:19:02.134923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.134889 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:02.139319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.139283 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:02.140566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.140526 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5px2t\" (UniqueName: \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.140862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.140834 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.145403 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.145372 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.170250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.170218 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2d8f0cae-dockercfg-mwcm9\"" Oct 31 00:19:02.177691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.177654 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:02.243152 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.243116 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5px2t\" (UniqueName: \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.243347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.243242 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb72b\" (UniqueName: \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.243347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.243298 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.243459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.243412 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.244088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.244062 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.261154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.261117 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5px2t\" (UniqueName: \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\") pod \"payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.354176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.354081 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb72b\" (UniqueName: \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.354467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.354448 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.379580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.379544 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.408678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.408640 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb72b\" (UniqueName: \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\") pod \"payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:02.481025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.480987 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:02.484652 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:02.484428 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:03.034905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.034874 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:03.036015 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:03.035989 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod58965b42_0e1c_480e_aa97_f23364bc0440.slice/crio-d6c3407d6b6fe6a2a560f0050b0db86173b557980afbf98de6c7674155751647 WatchSource:0}: Error finding container d6c3407d6b6fe6a2a560f0050b0db86173b557980afbf98de6c7674155751647: Status 404 returned error can't find the container with id d6c3407d6b6fe6a2a560f0050b0db86173b557980afbf98de6c7674155751647 Oct 31 00:19:03.083203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.083171 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:03.088599 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:03.088573 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod996d79da_5e17_45a7_ac7f_5d07cc527bc2.slice/crio-26168aba6b808f85b81a510bbab3da2617858b7a43fb3bd0fd1e2928c3c2867a WatchSource:0}: Error finding container 26168aba6b808f85b81a510bbab3da2617858b7a43fb3bd0fd1e2928c3c2867a: Status 404 returned error can't find the container with id 26168aba6b808f85b81a510bbab3da2617858b7a43fb3bd0fd1e2928c3c2867a Oct 31 00:19:03.165264 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:03.165227 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:19:03.574025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.573989 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" event={"ID":"996d79da-5e17-45a7-ac7f-5d07cc527bc2","Type":"ContainerStarted","Data":"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0"} Oct 31 00:19:03.574239 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.574055 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" event={"ID":"996d79da-5e17-45a7-ac7f-5d07cc527bc2","Type":"ContainerStarted","Data":"26168aba6b808f85b81a510bbab3da2617858b7a43fb3bd0fd1e2928c3c2867a"} Oct 31 00:19:03.574239 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.574099 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:03.575870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.575839 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" event={"ID":"58965b42-0e1c-480e-aa97-f23364bc0440","Type":"ContainerStarted","Data":"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d"} Oct 31 00:19:03.576306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.575883 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" event={"ID":"58965b42-0e1c-480e-aa97-f23364bc0440","Type":"ContainerStarted","Data":"d6c3407d6b6fe6a2a560f0050b0db86173b557980afbf98de6c7674155751647"} Oct 31 00:19:03.576306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.575935 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:03.602932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.602882 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" podStartSLOduration=1.60286745 podStartE2EDuration="1.60286745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:03.59962262 +0000 UTC m=+1465.632718684" watchObservedRunningTime="2026-10-31 00:19:03.60286745 +0000 UTC m=+1465.635963520" Oct 31 00:19:03.627114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:03.627060 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" podStartSLOduration=1.6270423470000002 podStartE2EDuration="1.627042347s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:03.623835152 +0000 UTC m=+1465.656931223" watchObservedRunningTime="2026-10-31 00:19:03.627042347 +0000 UTC m=+1465.660138417" Oct 31 00:19:05.526804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.526751 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:05.531527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.531507 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.548036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.548008 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:05.548756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.548729 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-3f561470\"" Oct 31 00:19:05.548869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.548733 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-3f561470-dockercfg-p7hxb\"" Oct 31 00:19:05.557799 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.557759 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.557955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.557929 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26n9n\" (UniqueName: \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.660808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.659585 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.660808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.659657 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26n9n\" (UniqueName: \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.668806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.668129 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.789868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.768527 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26n9n\" (UniqueName: \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\") pod \"payload-processing-e2e-ait-3f561470-5799b9648c-n6hft\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:05.852382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:05.852090 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:06.249712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.249475 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:06.249945 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:06.249911 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14dadb31_9b63_4c5e_8182_67278d1685c7.slice/crio-15373d6de5d19825072c308386db4670ee8db60c6414619d34e9565f2d20564d WatchSource:0}: Error finding container 15373d6de5d19825072c308386db4670ee8db60c6414619d34e9565f2d20564d: Status 404 returned error can't find the container with id 15373d6de5d19825072c308386db4670ee8db60c6414619d34e9565f2d20564d Oct 31 00:19:06.599488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.599401 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" event={"ID":"14dadb31-9b63-4c5e-8182-67278d1685c7","Type":"ContainerStarted","Data":"e9b2005c81594415c6be75c6e76d520a09d89acef57922429a951bf7f76bfdc3"} Oct 31 00:19:06.599488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.599455 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" event={"ID":"14dadb31-9b63-4c5e-8182-67278d1685c7","Type":"ContainerStarted","Data":"15373d6de5d19825072c308386db4670ee8db60c6414619d34e9565f2d20564d"} Oct 31 00:19:06.599909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.599622 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:06.621150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.621105 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" podStartSLOduration=1.621090696 podStartE2EDuration="1.621090696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:06.619027548 +0000 UTC m=+1468.652123619" watchObservedRunningTime="2026-10-31 00:19:06.621090696 +0000 UTC m=+1468.654186765" Oct 31 00:19:06.913890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.913770 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:19:06.918504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.918481 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.936589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.935844 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2854448d-openshift-default-dockercfg-m4xqs\"" Oct 31 00:19:06.940342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.940312 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:19:06.951552 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.951523 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wnwk\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.951672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.951594 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.951718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.951686 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.951778 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.951761 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.951778 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.951887 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.952037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.952018 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.952100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.952051 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.952156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.952128 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:06.952213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:06.952175 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.194861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.170029 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.198878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.177206 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.212894 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213086 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wnwk\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213175 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213237 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213307 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213429 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213582 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.213604 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.217807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.214474 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.219377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.218813 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.219377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.219111 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.219377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.219325 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.219928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.219907 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.236366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.236335 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.269897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.267462 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wnwk\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.279397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.279328 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\") pod \"e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.532842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.532723 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:07.827851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.827820 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:19:07.828240 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:07.828218 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec4925ae_5ad5_48f0_9a03_526b8fe2f573.slice/crio-81e2499c3a71109b1c4c88691d375064a49f9bcea7f1d8952df8c262e4343173 WatchSource:0}: Error finding container 81e2499c3a71109b1c4c88691d375064a49f9bcea7f1d8952df8c262e4343173: Status 404 returned error can't find the container with id 81e2499c3a71109b1c4c88691d375064a49f9bcea7f1d8952df8c262e4343173 Oct 31 00:19:07.830932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.830898 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:07.830999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.830967 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:07.831031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:07.831005 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:08.734237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:08.734193 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerStarted","Data":"977b78001590d50f6a14dd857fc283b7e5cdb0db046503e124c38322a918cb8f"} Oct 31 00:19:08.734446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:08.734258 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerStarted","Data":"81e2499c3a71109b1c4c88691d375064a49f9bcea7f1d8952df8c262e4343173"} Oct 31 00:19:08.762768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:08.762724 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podStartSLOduration=2.762710266 podStartE2EDuration="2.762710266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:08.760201292 +0000 UTC m=+1470.793297359" watchObservedRunningTime="2026-10-31 00:19:08.762710266 +0000 UTC m=+1470.795806335" Oct 31 00:19:09.533225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:09.533193 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:09.538500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:09.538477 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:09.860291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:09.810384 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:09.860291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:09.819104 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:19:10.794810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:10.793890 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:10.802809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:10.798878 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.180989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.180889 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.274814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.270978 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.281962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.281914 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.282601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.282563 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:11.283060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.283037 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:11.411834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.407884 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:11.428084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.428041 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.570994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.570955 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:11.571259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.571228 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.62:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.809811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.809209 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:11.809811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.809616 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:11.810411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.810207 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:19:11.810411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.810270 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:19:11.845220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:11.844909 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.053335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.052676 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.191012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.189930 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.274207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.272686 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:12.351812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.292915 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:12.351812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.293342 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:12.351812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.293622 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:12.494864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.413398 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:12.604810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.588988 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:19:12.605153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.605125 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:19:12.605341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.605325 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:19:12.605485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.605471 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:19:12.605623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.605611 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:19:12.605758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.605746 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:19:12.819876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.809633 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:19:12.819876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.809805 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:12.839634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.839596 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:12.839932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:12.839900 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.223851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.223802 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Oct 31 00:19:13.313924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.313855 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:13.396814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.392091 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:13.427810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.424242 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:13.457610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.435675 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.462809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.435697 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:13.494872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.463293 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:13.494872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.463355 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.71:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.584814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.582902 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.685831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.684817 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.701085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.701042 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:13.752872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.751961 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.75:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:13.786812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.753321 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:19:13.964811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:13.964224 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.006946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.006830 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.028829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.025095 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:19:14.028829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.025454 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:14.028829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.025492 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.028829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.026453 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:19:14.062042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.048835 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:14.062042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.048904 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.76:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.399112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.399007 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:14.497810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.464272 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.528812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.477395 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:14.529446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.477440 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:14.604925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.604869 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.650876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.644007 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:14.698809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.685677 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:14.793207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.793154 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:14.970942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:14.970886 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.029854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.013955 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.030295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.030235 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:19:15.177837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.176917 6234 patch_prober.go:29] interesting pod/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:15.177837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.176992 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.45:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.193836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.178265 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:15.194201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.194117 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.44:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.273881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.228080 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.416923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.402896 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.524563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.523325 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:15.588812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.563366 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.594840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.594770 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:15.616586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.616538 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:15.802032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.801928 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:19:15.994281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:15.992906 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.041828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.014436 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.041828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.039890 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.447813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.409977 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.545925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.545319 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.654759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.622446 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:16.654759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.648199 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:16.654759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.648270 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.81:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.654759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.648701 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:16.654759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.648815 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:16.765879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.761695 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": context deadline exceeded" Oct 31 00:19:16.810221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:16.810167 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.013810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.013642 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.042984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.042876 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:19:17.118008 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:17.117969 6234 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.035s" Oct 31 00:19:17.126388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.015820 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.150232 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.150176 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:19:17.279197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.279140 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.569811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.509886 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.702123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.663987 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.702123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.664338 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:17.702123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:17.664532 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.065409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.065346 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.156807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.151817 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:18.156807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.151897 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.83:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.176460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.176412 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.196598 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.196533 6234 patch_prober.go:29] interesting pod/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:19:18.232113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.232062 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.310237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.198036 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:18.319872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.197301 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.623218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.623104 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": context deadline exceeded" Oct 31 00:19:18.699627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.699577 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:18.784811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.762136 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:18.798376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.798310 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:18.843008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.805333 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:18.854852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:18.805383 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:19.077809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.067150 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:19.185029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.184979 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:19.237825 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:19.224488 6234 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.133s" Oct 31 00:19:19.238124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.238098 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:19:19.311397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.311312 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:19.314623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.314588 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:19:19.390196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.390096 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:19.562169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.562140 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:19:19.563021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.559333 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2ghzw\" (UniqueName: \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.564196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.564174 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:19:19.564413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.564402 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:19:19.564866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.564337 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.576221 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.576362 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.576441 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.576499 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.522944 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.582762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.580206 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.592414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.590718 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2854448d\"" Oct 31 00:19:19.592844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.592818 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2854448d-dockercfg-lb2pj\"" Oct 31 00:19:19.606811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.606147 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:19:19.609380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.608141 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:19:19.675693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.675613 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bngs4\" (UniqueName: \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.678404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.678374 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.682302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.682275 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.682894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.682853 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2ghzw\" (UniqueName: \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.683137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.683111 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.701553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.701523 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:19:19.705737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.705710 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2ghzw\" (UniqueName: \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\") pod \"payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:19.717488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.717461 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:19:19.784475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.784433 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.784919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.784893 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bngs4\" (UniqueName: \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.785456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.785435 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.806526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.806488 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bngs4\" (UniqueName: \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\") pod \"payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:19.959753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:19.959718 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:20.000475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.000427 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:20.303478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.303436 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:19:20.304128 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:20.304100 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode006a0ec_3727_4cd2_bd24_be3ca19ae0b8.slice/crio-104499ddf40f09fc4f41bae736052ab15ae6459e73b75b59ff5fd41b4dc7797b WatchSource:0}: Error finding container 104499ddf40f09fc4f41bae736052ab15ae6459e73b75b59ff5fd41b4dc7797b: Status 404 returned error can't find the container with id 104499ddf40f09fc4f41bae736052ab15ae6459e73b75b59ff5fd41b4dc7797b Oct 31 00:19:20.343274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.343244 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:19:20.345068 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:20.345022 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80b9c727_469f_4b4d_8fa5_c3a732f8145d.slice/crio-c66a3cf6f01c7e4eb2db43321144c20313d90b735c8be553b00f7a73b303ca76 WatchSource:0}: Error finding container c66a3cf6f01c7e4eb2db43321144c20313d90b735c8be553b00f7a73b303ca76: Status 404 returned error can't find the container with id c66a3cf6f01c7e4eb2db43321144c20313d90b735c8be553b00f7a73b303ca76 Oct 31 00:19:20.827405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.827331 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" event={"ID":"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8","Type":"ContainerStarted","Data":"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897"} Oct 31 00:19:20.827607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.827590 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:20.827730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.827632 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" event={"ID":"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8","Type":"ContainerStarted","Data":"104499ddf40f09fc4f41bae736052ab15ae6459e73b75b59ff5fd41b4dc7797b"} Oct 31 00:19:20.829483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.829458 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" event={"ID":"80b9c727-469f-4b4d-8fa5-c3a732f8145d","Type":"ContainerStarted","Data":"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271"} Oct 31 00:19:20.829620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.829496 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" event={"ID":"80b9c727-469f-4b4d-8fa5-c3a732f8145d","Type":"ContainerStarted","Data":"c66a3cf6f01c7e4eb2db43321144c20313d90b735c8be553b00f7a73b303ca76"} Oct 31 00:19:20.829620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.829607 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:20.978761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.975887 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" podStartSLOduration=3.975865245 podStartE2EDuration="3.975865245s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:20.903279611 +0000 UTC m=+1482.936375681" watchObservedRunningTime="2026-10-31 00:19:20.975865245 +0000 UTC m=+1483.008961321" Oct 31 00:19:20.978761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:20.976138 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" podStartSLOduration=3.976126419 podStartE2EDuration="3.976126419s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:20.955373102 +0000 UTC m=+1482.988469176" watchObservedRunningTime="2026-10-31 00:19:20.976126419 +0000 UTC m=+1483.009222489" Oct 31 00:19:29.367678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.367644 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:19:29.372375 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.372356 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.384283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.382094 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-2854448d-serving-cert\"" Oct 31 00:19:29.389481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.389451 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:19:29.535235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.535198 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.535378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.535333 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.535552 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.535529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.535590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.535577 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26lrc\" (UniqueName: \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.644858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.644274 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.644858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.644341 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26lrc\" (UniqueName: \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.644858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.644539 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.644858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.644653 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.663810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.653168 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.684823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.683460 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26lrc\" (UniqueName: \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.741590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.724450 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.756987 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.756713 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\") pod \"maas-api-e2e-ait-2854448d-7c77cd545d-vh2np\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:29.994488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:29.993242 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:30.495437 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:30.495366 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2e2000d_a2e5_4dd1_a102_7249f1d31970.slice/crio-9b202d6467d51a3f4d53aa60e3349c72969753b5d131ae9347a347eb47474ae5 WatchSource:0}: Error finding container 9b202d6467d51a3f4d53aa60e3349c72969753b5d131ae9347a347eb47474ae5: Status 404 returned error can't find the container with id 9b202d6467d51a3f4d53aa60e3349c72969753b5d131ae9347a347eb47474ae5 Oct 31 00:19:30.496236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:30.496214 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:19:30.932828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:30.932724 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" event={"ID":"e2e2000d-a2e5-4dd1-a102-7249f1d31970","Type":"ContainerStarted","Data":"9b202d6467d51a3f4d53aa60e3349c72969753b5d131ae9347a347eb47474ae5"} Oct 31 00:19:31.840913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:31.840881 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:19:31.841314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:31.840962 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:19:31.946846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:31.946809 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" event={"ID":"e2e2000d-a2e5-4dd1-a102-7249f1d31970","Type":"ContainerStarted","Data":"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc"} Oct 31 00:19:31.947043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:31.947031 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:33.163460 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:33.163416 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:19:37.997808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:37.995884 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:19:38.047737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:38.047526 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" podStartSLOduration=8.583432981 podStartE2EDuration="9.047506058s" totalImagesPullingTime="464.073077ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:31.980009334 +0000 UTC m=+1494.013105404" watchObservedRunningTime="2026-10-31 00:19:38.047506058 +0000 UTC m=+1500.080602128" Oct 31 00:19:45.193815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.190386 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:45.494865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.491878 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:45.531941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.531892 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:45.632919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.631881 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:45.633231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.633183 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:45.825976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:45.823890 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.420918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.420848 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:19:46.498215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.498154 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.499363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.499326 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:19:46.558816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.545921 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.558816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.558104 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:19:46.642377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.642320 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.644536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.644510 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:19:46.702848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.696885 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.792812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.734247 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:19:46.795330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.795283 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.846812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.816905 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:19:46.879868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.830887 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:46.891810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:46.881382 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:19:47.077915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.071623 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.452873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.430752 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.561810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.545874 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.562023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.561946 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.563812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.563092 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.563812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.563704 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:19:47.714269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.714175 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:47.759823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.734068 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.768408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.756537 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.845818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.830582 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/echo-simulator-kserve-64b6f8f96c-5bfgv" podUID="354612ef-bb88-4c36-b5fe-caf0ca868ce7" containerName="main" probeResult="failure" output="Get \"https://10.134.0.42:8000/ready\": context deadline exceeded" Oct 31 00:19:47.920559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.850072 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:47.920559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.850434 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:19:47.920559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:47.883249 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.088955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.072600 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.088955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.074135 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:19:48.437136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.436468 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.437136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.436739 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.635843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.568061 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:48.635843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.606165 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:19:48.635843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.607075 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:19:48.801974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.793116 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.801974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.793429 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:48.801974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.793616 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:48.867939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.867887 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:48.898467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:48.884150 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.245878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.145714 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:49.543975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.441304 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.636809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.631699 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:49.636809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.635527 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:19:49.809480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.796844 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:19:49.809480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.797122 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:49.809480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.797301 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:49.891849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.875450 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:49.926812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.907817 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.006813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:49.976187 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": context deadline exceeded" Oct 31 00:19:50.156072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.155971 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:50.562955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.562903 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.579859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.579809 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.670441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.659695 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:50.670441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.659858 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.849875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.811349 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.849875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.834935 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.924343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.924300 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.925855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.925813 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:50.937410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:50.927421 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:19:51.281811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.280072 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:51.372397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.372361 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:51.372758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.372739 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:51.651180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.567294 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:51.651756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.584658 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:51.685267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.685225 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:51.687174 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.687144 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:19:51.812931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.812873 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:19:51.893943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.841872 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:51.895902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.841912 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 31 00:19:51.927131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.927037 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:51.930086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.930034 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:51.960086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:51.960025 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:52.168809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.057161 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:52.333815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.285907 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:19:52.642859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.642622 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:19:52.642859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.642701 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:19:52.642859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.642736 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:19:52.695770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.695716 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:19:52.927987 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.927882 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:53.019958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.930994 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:19:53.028638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:52.983356 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:19:53.081383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.080083 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:19:53.082884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.082861 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:19:53.108926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.107907 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:19:53.113808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.111392 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:19:53.115158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.114320 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:19:53.115521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:53.115502 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:19:56.374626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.374577 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" podUID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" containerName="payload-processing" containerID="cri-o://8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0" gracePeriod=25 Oct 31 00:19:56.656957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.652533 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" podUID="58965b42-0e1c-480e-aa97-f23364bc0440" containerName="payload-pre-processing" containerID="cri-o://4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d" gracePeriod=25 Oct 31 00:19:56.766057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.766031 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:56.861126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.861082 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\") pod \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " Oct 31 00:19:56.861317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.861255 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\" (UniqueName: \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\") pod \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\" (UID: \"996d79da-5e17-45a7-ac7f-5d07cc527bc2\") " Oct 31 00:19:56.861861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.861823 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume" pod "996d79da-5e17-45a7-ac7f-5d07cc527bc2" (UID: "996d79da-5e17-45a7-ac7f-5d07cc527bc2"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:56.863102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.863079 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/996d79da-5e17-45a7-ac7f-5d07cc527bc2-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:56.864537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.864497 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t" pod "996d79da-5e17-45a7-ac7f-5d07cc527bc2" (UID: "996d79da-5e17-45a7-ac7f-5d07cc527bc2"). InnerVolumeSpecName "kube-api-access-5px2t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:56.928579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.928494 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" podUID="14dadb31-9b63-4c5e-8182-67278d1685c7" containerName="payload-processing" containerID="cri-o://e9b2005c81594415c6be75c6e76d520a09d89acef57922429a951bf7f76bfdc3" gracePeriod=25 Oct 31 00:19:56.964743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:56.964712 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5px2t\" (UniqueName: \"kubernetes.io/projected/996d79da-5e17-45a7-ac7f-5d07cc527bc2-kube-api-access-5px2t\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:57.104139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.102726 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:57.173818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.172835 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\") pod \"58965b42-0e1c-480e-aa97-f23364bc0440\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " Oct 31 00:19:57.173818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.172974 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\" (UniqueName: \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\") pod \"58965b42-0e1c-480e-aa97-f23364bc0440\" (UID: \"58965b42-0e1c-480e-aa97-f23364bc0440\") " Oct 31 00:19:57.192948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.182079 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume" pod "58965b42-0e1c-480e-aa97-f23364bc0440" (UID: "58965b42-0e1c-480e-aa97-f23364bc0440"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:57.193390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.193245 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b" pod "58965b42-0e1c-480e-aa97-f23364bc0440" (UID: "58965b42-0e1c-480e-aa97-f23364bc0440"). InnerVolumeSpecName "kube-api-access-wb72b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:57.206118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.205700 6234 generic.go:356] "Generic (PLEG): container finished" podID="58965b42-0e1c-480e-aa97-f23364bc0440" containerID="4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d" exitCode=0 Oct 31 00:19:57.206118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.205883 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" event={"ID":"58965b42-0e1c-480e-aa97-f23364bc0440","Type":"ContainerDied","Data":"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d"} Oct 31 00:19:57.206118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.205939 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" event={"ID":"58965b42-0e1c-480e-aa97-f23364bc0440","Type":"ContainerDied","Data":"d6c3407d6b6fe6a2a560f0050b0db86173b557980afbf98de6c7674155751647"} Oct 31 00:19:57.206118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.205961 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6" Oct 31 00:19:57.206118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.205993 6234 scope.go:122] "RemoveContainer" containerID="4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d" Oct 31 00:19:57.213429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.210699 6234 generic.go:356] "Generic (PLEG): container finished" podID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" containerID="8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0" exitCode=0 Oct 31 00:19:57.213429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.210795 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" event={"ID":"996d79da-5e17-45a7-ac7f-5d07cc527bc2","Type":"ContainerDied","Data":"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0"} Oct 31 00:19:57.213429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.210843 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" event={"ID":"996d79da-5e17-45a7-ac7f-5d07cc527bc2","Type":"ContainerDied","Data":"26168aba6b808f85b81a510bbab3da2617858b7a43fb3bd0fd1e2928c3c2867a"} Oct 31 00:19:57.213429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.210912 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x" Oct 31 00:19:57.217838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.217600 6234 generic.go:356] "Generic (PLEG): container finished" podID="14dadb31-9b63-4c5e-8182-67278d1685c7" containerID="e9b2005c81594415c6be75c6e76d520a09d89acef57922429a951bf7f76bfdc3" exitCode=0 Oct 31 00:19:57.217838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.217763 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" event={"ID":"14dadb31-9b63-4c5e-8182-67278d1685c7","Type":"ContainerDied","Data":"e9b2005c81594415c6be75c6e76d520a09d89acef57922429a951bf7f76bfdc3"} Oct 31 00:19:57.238628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.236873 6234 scope.go:122] "RemoveContainer" containerID="4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d" Oct 31 00:19:57.238628 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:57.237256 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d\": container with ID starting with 4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d not found: ID does not exist" containerID="4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d" Oct 31 00:19:57.238628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.237291 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d"} err="failed to get container status \"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d\": rpc error: code = NotFound desc = could not find container \"4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d\": container with ID starting with 4cfb86c852d53b0a896c3512bccbbf20299858708584a8e436f894e5a668d10d not found: ID does not exist" Oct 31 00:19:57.238628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.237315 6234 scope.go:122] "RemoveContainer" containerID="8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0" Oct 31 00:19:57.239037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.238645 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:19:57.239903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.239878 6234 cpu_manager.go:414] "containerMap: removing container" podUID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" containerName="payload-processing" Oct 31 00:19:57.240024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.239905 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" containerName="payload-processing" Oct 31 00:19:57.240024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.239938 6234 cpu_manager.go:414] "containerMap: removing container" podUID="58965b42-0e1c-480e-aa97-f23364bc0440" containerName="payload-pre-processing" Oct 31 00:19:57.240024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.239946 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58965b42-0e1c-480e-aa97-f23364bc0440" containerName="payload-pre-processing" Oct 31 00:19:57.240172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.240148 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" containerName="payload-processing" Oct 31 00:19:57.240172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.240164 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="58965b42-0e1c-480e-aa97-f23364bc0440" containerName="payload-pre-processing" Oct 31 00:19:57.249361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.248182 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.265755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.262207 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:57.271647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.271608 6234 scope.go:122] "RemoveContainer" containerID="8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0" Oct 31 00:19:57.272723 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:19:57.272660 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0\": container with ID starting with 8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0 not found: ID does not exist" containerID="8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0" Oct 31 00:19:57.272887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.272694 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0"} err="failed to get container status \"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0\": rpc error: code = NotFound desc = could not find container \"8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0\": container with ID starting with 8ff1cf39655de46b636893c08ed4d8d13a9a87d170d1cc1f5a811dc5a3e4c5f0 not found: ID does not exist" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274853 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274890 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274925 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274955 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274979 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.274996 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bc8bw\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.275067 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.275107 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.275180 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.275234 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/58965b42-0e1c-480e-aa97-f23364bc0440-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:57.275352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.275244 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb72b\" (UniqueName: \"kubernetes.io/projected/58965b42-0e1c-480e-aa97-f23364bc0440-kube-api-access-wb72b\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:57.276497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.276472 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-2d8f0cae-openshift-default-dockercfg-wt5wq\"" Oct 31 00:19:57.281830 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.281775 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:19:57.287299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.287275 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2d8f0cae-7bcb446fbc-n7ng6"] Oct 31 00:19:57.293604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.293581 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:57.303959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.303931 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2d8f0cae-6bd4cf7b77-fpr6x"] Oct 31 00:19:57.376497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376460 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376530 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376558 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376603 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376642 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376675 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376698 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bc8bw\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376836 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.376993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.376897 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.377482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.377354 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.378018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.377920 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.378216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.378196 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.378821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.378465 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.378821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.378570 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.381450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.381423 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.383982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.383942 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.393535 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.393510 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.394386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.394363 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bc8bw\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\") pod \"e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.436000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.435975 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:57.477771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.477660 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\" (UniqueName: \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\") pod \"14dadb31-9b63-4c5e-8182-67278d1685c7\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " Oct 31 00:19:57.477771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.477725 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\") pod \"14dadb31-9b63-4c5e-8182-67278d1685c7\" (UID: \"14dadb31-9b63-4c5e-8182-67278d1685c7\") " Oct 31 00:19:57.478457 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.478430 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume" pod "14dadb31-9b63-4c5e-8182-67278d1685c7" (UID: "14dadb31-9b63-4c5e-8182-67278d1685c7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:19:57.478691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.478672 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/14dadb31-9b63-4c5e-8182-67278d1685c7-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:57.480478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.480451 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n" pod "14dadb31-9b63-4c5e-8182-67278d1685c7" (UID: "14dadb31-9b63-4c5e-8182-67278d1685c7"). InnerVolumeSpecName "kube-api-access-26n9n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:19:57.580138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.580101 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26n9n\" (UniqueName: \"kubernetes.io/projected/14dadb31-9b63-4c5e-8182-67278d1685c7-kube-api-access-26n9n\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:19:57.606643 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.606606 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:57.914927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.914844 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:19:57.919111 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:19:57.919082 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poded8de31f_6250_40b9_a94d_e37b2c7c9c54.slice/crio-9ae2c45410bd6cad24cb2fa23ebbd31531ca1e9f9e1698f6f4503835441f32e0 WatchSource:0}: Error finding container 9ae2c45410bd6cad24cb2fa23ebbd31531ca1e9f9e1698f6f4503835441f32e0: Status 404 returned error can't find the container with id 9ae2c45410bd6cad24cb2fa23ebbd31531ca1e9f9e1698f6f4503835441f32e0 Oct 31 00:19:57.922071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.922035 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:57.922177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.922110 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:57.922177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:57.922141 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:19:58.086613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.086576 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58965b42-0e1c-480e-aa97-f23364bc0440" path="/var/lib/kubelet/pods/58965b42-0e1c-480e-aa97-f23364bc0440/volumes" Oct 31 00:19:58.087184 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.087163 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="996d79da-5e17-45a7-ac7f-5d07cc527bc2" path="/var/lib/kubelet/pods/996d79da-5e17-45a7-ac7f-5d07cc527bc2/volumes" Oct 31 00:19:58.232114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.232071 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" event={"ID":"ed8de31f-6250-40b9-a94d-e37b2c7c9c54","Type":"ContainerStarted","Data":"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4"} Oct 31 00:19:58.232322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.232138 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" event={"ID":"ed8de31f-6250-40b9-a94d-e37b2c7c9c54","Type":"ContainerStarted","Data":"9ae2c45410bd6cad24cb2fa23ebbd31531ca1e9f9e1698f6f4503835441f32e0"} Oct 31 00:19:58.234828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.234777 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" event={"ID":"14dadb31-9b63-4c5e-8182-67278d1685c7","Type":"ContainerDied","Data":"15373d6de5d19825072c308386db4670ee8db60c6414619d34e9565f2d20564d"} Oct 31 00:19:58.234985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.234846 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft" Oct 31 00:19:58.234985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.234860 6234 scope.go:122] "RemoveContainer" containerID="e9b2005c81594415c6be75c6e76d520a09d89acef57922429a951bf7f76bfdc3" Oct 31 00:19:58.351808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.343310 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podStartSLOduration=1.343289578 podStartE2EDuration="1.343289578s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:19:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:19:58.332281393 +0000 UTC m=+1520.365377487" watchObservedRunningTime="2026-10-31 00:19:58.343289578 +0000 UTC m=+1520.376385661" Oct 31 00:19:58.407959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.407915 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:58.423216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.423183 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-3f561470-5799b9648c-n6hft"] Oct 31 00:19:58.608386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:58.608301 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:19:59.626813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:59.624916 6234 patch_prober.go:29] interesting pod/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:19:59.626813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:19:59.625017 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:20:00.085920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:00.085883 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="14dadb31-9b63-4c5e-8182-67278d1685c7" path="/var/lib/kubelet/pods/14dadb31-9b63-4c5e-8182-67278d1685c7/volumes" Oct 31 00:20:00.609241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:00.609150 6234 patch_prober.go:29] interesting pod/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:20:00.609241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:00.609218 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:20:01.287488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.287457 6234 generic.go:356] "Generic (PLEG): container finished" podID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerID="977b78001590d50f6a14dd857fc283b7e5cdb0db046503e124c38322a918cb8f" exitCode=0 Oct 31 00:20:01.287900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.287528 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerDied","Data":"977b78001590d50f6a14dd857fc283b7e5cdb0db046503e124c38322a918cb8f"} Oct 31 00:20:01.288102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.288081 6234 scope.go:122] "RemoveContainer" containerID="977b78001590d50f6a14dd857fc283b7e5cdb0db046503e124c38322a918cb8f" Oct 31 00:20:01.289763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.289724 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:01.289950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.289846 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:01.289950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.289885 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:01.608463 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.608384 6234 patch_prober.go:29] interesting pod/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:20:01.608463 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:01.608437 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:20:02.307381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.307341 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerStarted","Data":"3c27689f3d3a576b7810460c4dedfd3cb3ab6a1680398132e7a4f1933dd8dbf5"} Oct 31 00:20:02.532945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.532907 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:02.533139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.533119 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:02.533200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.533180 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:02.608333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.608236 6234 patch_prober.go:29] interesting pod/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" start-of-body= Oct 31 00:20:02.608333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:02.608299 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.96:15021/healthz/ready\": dial tcp 10.134.0.96:15021: connect: connection refused" Oct 31 00:20:03.335811 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:03.335071 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:20:03.509173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:03.508993 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:20:03.509387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:03.509272 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" containerID="cri-o://e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4" gracePeriod=30 Oct 31 00:20:03.533615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:03.533293 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:03.533615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:03.533339 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:04.533941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:04.533906 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:04.534434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:04.533973 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:05.122618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.122587 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:20:05.123395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.123373 6234 cpu_manager.go:414] "containerMap: removing container" podUID="14dadb31-9b63-4c5e-8182-67278d1685c7" containerName="payload-processing" Oct 31 00:20:05.123395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.123391 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="14dadb31-9b63-4c5e-8182-67278d1685c7" containerName="payload-processing" Oct 31 00:20:05.123596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.123532 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="14dadb31-9b63-4c5e-8182-67278d1685c7" containerName="payload-processing" Oct 31 00:20:05.127603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.127583 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.152985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.152954 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-f71e7a13-gw-openshift-default-dockercfg-jt6zg\"" Oct 31 00:20:05.200727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.200686 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:20:05.308424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.308377 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.308621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.308544 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.308742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.308720 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.308867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.308848 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.308934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.308907 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mtpmm\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.309074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.309053 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.309119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.309082 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.309260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.309240 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.309325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.309292 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.414202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.414044 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mtpmm\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.416384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.416356 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.430460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.430430 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.431041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.431013 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.431217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.431202 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.431356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.431341 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.431536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.431522 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.435255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.435160 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.438102 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.438213 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.438675 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.441294 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.441507 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.441747 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.452558 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mtpmm\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.463819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.458096 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.478808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.470537 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.510809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.504424 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\") pod \"e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:05.547842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.545910 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:05.547842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.545976 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:05.832982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:05.832718 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:06.506290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.506254 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:20:06.519235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.519186 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:06.519394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.519288 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:06.519394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.519336 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:06.533233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.533199 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:06.533367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:06.533246 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:07.550807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.534481 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:07.556811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.551399 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:07.718543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.718498 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" event={"ID":"38a9b733-9eae-40c9-bca2-77b56be7f749","Type":"ContainerStarted","Data":"d12190c4003252c4ca1a7e4ca68167f7b56cdf5701210804a71c8c3c8f74622f"} Oct 31 00:20:07.771823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.735916 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" event={"ID":"38a9b733-9eae-40c9-bca2-77b56be7f749","Type":"ContainerStarted","Data":"3206337f035de271f7331aaff45114f7e852d778dcec567e60d104e90ce514d5"} Oct 31 00:20:07.845859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.845696 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:07.847345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.847287 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" podStartSLOduration=2.847267473 podStartE2EDuration="2.847267473s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:07.759945044 +0000 UTC m=+1529.793041140" watchObservedRunningTime="2026-10-31 00:20:07.847267473 +0000 UTC m=+1529.880363543" Oct 31 00:20:07.868099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.868065 6234 patch_prober.go:29] interesting pod/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" start-of-body= Oct 31 00:20:07.869117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:07.869084 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.97:15021/healthz/ready\": dial tcp 10.134.0.97:15021: connect: connection refused" Oct 31 00:20:08.185645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:08.185555 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:20:08.204290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:08.204260 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:20:08.533613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:08.533582 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:08.533850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:08.533631 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:09.021591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.021557 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:09.533313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.533280 6234 patch_prober.go:29] interesting pod/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body= Oct 31 00:20:09.533487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.533333 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" Oct 31 00:20:09.800000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.799795 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:09.902918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.902848 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:09.904648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:09.904625 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:10.538668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.538633 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:10.540024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.540003 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:10.624178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.624106 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:20:10.727177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727147 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727216 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727372 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727488 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727533 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727593 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727740 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.727932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.727886 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\") pod \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\" (UID: \"ed8de31f-6250-40b9-a94d-e37b2c7c9c54\") " Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.728601 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.728964 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729370 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729481 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729825 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729858 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729872 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.729909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.729886 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.731355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.731320 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:10.733029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.733004 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "kube-api-access-bc8bw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:10.733136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.733109 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:10.733977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.733955 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:10.736039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.736010 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo" pod "ed8de31f-6250-40b9-a94d-e37b2c7c9c54" (UID: "ed8de31f-6250-40b9-a94d-e37b2c7c9c54"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:20:10.755876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.755846 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:10.757113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.757091 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" Oct 31 00:20:10.757167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.757119 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" Oct 31 00:20:10.757339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.757325 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerName="istio-proxy" Oct 31 00:20:10.762048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.762021 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:10.762181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.762160 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:10.766625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.766603 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:10.778843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.778774 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:10.783359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.783275 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-f71e7a13\"" Oct 31 00:20:10.783472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.783430 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-f71e7a13-dockercfg-k4k65\"" Oct 31 00:20:10.783986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.783965 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:10.841951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.841914 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t2qwr\" (UniqueName: \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:10.842638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.842576 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:10.842923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.842903 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m57v7\" (UniqueName: \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:10.843239 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.843214 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:10.843570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.843549 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.843688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.843674 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.843818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.843801 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.843929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.843915 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.844024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.844009 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bc8bw\" (UniqueName: \"kubernetes.io/projected/ed8de31f-6250-40b9-a94d-e37b2c7c9c54-kube-api-access-bc8bw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:10.963610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.963552 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:11.033828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:10.964672 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:11.033828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.015276 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m57v7\" (UniqueName: \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:11.087834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.050313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:11.087834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.050502 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t2qwr\" (UniqueName: \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:11.087834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.061022 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:11.158746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.158715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t2qwr\" (UniqueName: \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\") pod \"payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:11.163245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.163221 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m57v7\" (UniqueName: \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\") pod \"payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:11.171314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.171285 6234 generic.go:356] "Generic (PLEG): container finished" podID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" containerID="e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4" exitCode=0 Oct 31 00:20:11.172503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.172481 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" Oct 31 00:20:11.176063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.176033 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" event={"ID":"ed8de31f-6250-40b9-a94d-e37b2c7c9c54","Type":"ContainerDied","Data":"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4"} Oct 31 00:20:11.176251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.176233 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5" event={"ID":"ed8de31f-6250-40b9-a94d-e37b2c7c9c54","Type":"ContainerDied","Data":"9ae2c45410bd6cad24cb2fa23ebbd31531ca1e9f9e1698f6f4503835441f32e0"} Oct 31 00:20:11.176384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.176372 6234 scope.go:122] "RemoveContainer" containerID="e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4" Oct 31 00:20:11.213678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.213592 6234 scope.go:122] "RemoveContainer" containerID="e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4" Oct 31 00:20:11.214583 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:11.214536 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4\": container with ID starting with e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4 not found: ID does not exist" containerID="e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4" Oct 31 00:20:11.214840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.214773 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4"} err="failed to get container status \"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4\": rpc error: code = NotFound desc = could not find container \"e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4\": container with ID starting with e06420fd16595be7170ff0862192a80bab29a729083e3a3899376f0fa3c86bb4 not found: ID does not exist" Oct 31 00:20:11.232357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.232322 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:20:11.248963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.248928 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2d8f0cae-openshift-default-6cbdffbb55-tzjq5"] Oct 31 00:20:11.381311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.381282 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:11.390399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.390376 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:11.680295 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:11.680253 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe069678_1488_40dc_afc9_30a761bf597d.slice/crio-b79199a1b87b57bd248aa7c10f8d02e6c533abb4091f633c3bb814ad60367d2d WatchSource:0}: Error finding container b79199a1b87b57bd248aa7c10f8d02e6c533abb4091f633c3bb814ad60367d2d: Status 404 returned error can't find the container with id b79199a1b87b57bd248aa7c10f8d02e6c533abb4091f633c3bb814ad60367d2d Oct 31 00:20:11.681240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.681197 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:11.714636 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:11.714589 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5380d776_9b8f_42d1_99b7_ad7a1109541e.slice/crio-ef10d9a119e99ca31ad51fcaf6f05f9d6575b5e44f73f7282b6bf72877fd672c WatchSource:0}: Error finding container ef10d9a119e99ca31ad51fcaf6f05f9d6575b5e44f73f7282b6bf72877fd672c: Status 404 returned error can't find the container with id ef10d9a119e99ca31ad51fcaf6f05f9d6575b5e44f73f7282b6bf72877fd672c Oct 31 00:20:11.718119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:11.718080 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:12.099897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.097095 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ed8de31f-6250-40b9-a94d-e37b2c7c9c54" path="/var/lib/kubelet/pods/ed8de31f-6250-40b9-a94d-e37b2c7c9c54/volumes" Oct 31 00:20:12.179698 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.179653 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" event={"ID":"5380d776-9b8f-42d1-99b7-ad7a1109541e","Type":"ContainerStarted","Data":"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e"} Oct 31 00:20:12.179900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.179724 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:12.179900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.179747 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" event={"ID":"5380d776-9b8f-42d1-99b7-ad7a1109541e","Type":"ContainerStarted","Data":"ef10d9a119e99ca31ad51fcaf6f05f9d6575b5e44f73f7282b6bf72877fd672c"} Oct 31 00:20:12.181507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.181472 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" event={"ID":"be069678-1488-40dc-afc9-30a761bf597d","Type":"ContainerStarted","Data":"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293"} Oct 31 00:20:12.181621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.181526 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" event={"ID":"be069678-1488-40dc-afc9-30a761bf597d","Type":"ContainerStarted","Data":"b79199a1b87b57bd248aa7c10f8d02e6c533abb4091f633c3bb814ad60367d2d"} Oct 31 00:20:12.181828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.181802 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:12.217695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.217649 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" podStartSLOduration=2.21763708 podStartE2EDuration="2.21763708s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:12.216226322 +0000 UTC m=+1534.249322393" watchObservedRunningTime="2026-10-31 00:20:12.21763708 +0000 UTC m=+1534.250733155" Oct 31 00:20:12.241182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:12.241135 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" podStartSLOduration=2.241122202 podStartE2EDuration="2.241122202s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:12.236870833 +0000 UTC m=+1534.269966913" watchObservedRunningTime="2026-10-31 00:20:12.241122202 +0000 UTC m=+1534.274218272" Oct 31 00:20:13.008498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.008462 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:13.013299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.013282 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.028480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.028439 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-15bcc310\"" Oct 31 00:20:13.031689 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.031655 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:13.036364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.036338 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:13.043730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.043704 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.051698 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.051670 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-15bcc310-dockercfg-pbcp8\"" Oct 31 00:20:13.053927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.053896 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:13.100892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.100857 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.101065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.101034 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsqj4\" (UniqueName: \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.187945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.187857 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" podUID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" containerName="payload-processing" containerID="cri-o://72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897" gracePeriod=25 Oct 31 00:20:13.212079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.212029 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" podUID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" containerName="payload-pre-processing" containerID="cri-o://0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271" gracePeriod=25 Oct 31 00:20:13.226568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.226535 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.267193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.249652 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.269897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.267924 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.269897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.268058 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsqj4\" (UniqueName: \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.269897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.268297 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxn8n\" (UniqueName: \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.321807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.302766 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsqj4\" (UniqueName: \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\") pod \"payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.345557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.345335 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:13.369948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.369623 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.369948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.369930 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxn8n\" (UniqueName: \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.373802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.371420 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.528929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.528848 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxn8n\" (UniqueName: \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\") pod \"payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.684145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.668664 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:13.860401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.860378 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:20:13.904648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.904023 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\") pod \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " Oct 31 00:20:13.904648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.904169 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\" (UniqueName: \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\") pod \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\" (UID: \"80b9c727-469f-4b4d-8fa5-c3a732f8145d\") " Oct 31 00:20:13.904648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.904551 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume" pod "80b9c727-469f-4b4d-8fa5-c3a732f8145d" (UID: "80b9c727-469f-4b4d-8fa5-c3a732f8145d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:13.905023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.905002 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/80b9c727-469f-4b4d-8fa5-c3a732f8145d-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:13.909799 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.908977 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:20:13.912325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.912298 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4" pod "80b9c727-469f-4b4d-8fa5-c3a732f8145d" (UID: "80b9c727-469f-4b4d-8fa5-c3a732f8145d"). InnerVolumeSpecName "kube-api-access-bngs4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:13.954258 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:13.954225 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c947ed8_29bc_4a1a_8cd2_c0f15f1ba2c8.slice/crio-8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4 WatchSource:0}: Error finding container 8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4: Status 404 returned error can't find the container with id 8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4 Oct 31 00:20:13.955472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:13.955446 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:14.005697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.005667 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\") pod \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " Oct 31 00:20:14.006898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.006047 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\" (UniqueName: \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\") pod \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\" (UID: \"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8\") " Oct 31 00:20:14.006898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.006144 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume" pod "e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" (UID: "e006a0ec-3727-4cd2-bd24-be3ca19ae0b8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:14.006898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.006880 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bngs4\" (UniqueName: \"kubernetes.io/projected/80b9c727-469f-4b4d-8fa5-c3a732f8145d-kube-api-access-bngs4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.006898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.006902 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.008327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.008308 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw" pod "e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" (UID: "e006a0ec-3727-4cd2-bd24-be3ca19ae0b8"). InnerVolumeSpecName "kube-api-access-2ghzw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:14.018522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.018499 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:14.024403 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:14.024377 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a0989e_c1d2_401d_92cb_64380491919e.slice/crio-5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581 WatchSource:0}: Error finding container 5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581: Status 404 returned error can't find the container with id 5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581 Oct 31 00:20:14.108722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.108696 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2ghzw\" (UniqueName: \"kubernetes.io/projected/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8-kube-api-access-2ghzw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.277811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.271609 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:20:14.277811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.271931 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" podUID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" containerName="maas-api" containerID="cri-o://4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc" gracePeriod=30 Oct 31 00:20:14.346510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.346419 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" event={"ID":"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8","Type":"ContainerStarted","Data":"c9d87a8323905e3976ed257c9434dc35b5939810def60e0085a5c80033e0372f"} Oct 31 00:20:14.346739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.346523 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" event={"ID":"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8","Type":"ContainerStarted","Data":"8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4"} Oct 31 00:20:14.346895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.346870 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:14.348525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.348492 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" event={"ID":"a9a0989e-c1d2-401d-92cb-64380491919e","Type":"ContainerStarted","Data":"41229088e7b3d144a0ca4399de7780116405dac24d2931a581ebf44d73046cab"} Oct 31 00:20:14.348645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.348561 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" event={"ID":"a9a0989e-c1d2-401d-92cb-64380491919e","Type":"ContainerStarted","Data":"5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581"} Oct 31 00:20:14.348853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.348830 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:14.350931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.350900 6234 generic.go:356] "Generic (PLEG): container finished" podID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" containerID="72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897" exitCode=0 Oct 31 00:20:14.351054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.350961 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" Oct 31 00:20:14.351054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.351000 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" event={"ID":"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8","Type":"ContainerDied","Data":"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897"} Oct 31 00:20:14.351155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.351060 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl" event={"ID":"e006a0ec-3727-4cd2-bd24-be3ca19ae0b8","Type":"ContainerDied","Data":"104499ddf40f09fc4f41bae736052ab15ae6459e73b75b59ff5fd41b4dc7797b"} Oct 31 00:20:14.351155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.351093 6234 scope.go:122] "RemoveContainer" containerID="72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897" Oct 31 00:20:14.352610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.352587 6234 generic.go:356] "Generic (PLEG): container finished" podID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" containerID="0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271" exitCode=0 Oct 31 00:20:14.352715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.352630 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" event={"ID":"80b9c727-469f-4b4d-8fa5-c3a732f8145d","Type":"ContainerDied","Data":"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271"} Oct 31 00:20:14.352715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.352646 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" Oct 31 00:20:14.352715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.352668 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms" event={"ID":"80b9c727-469f-4b4d-8fa5-c3a732f8145d","Type":"ContainerDied","Data":"c66a3cf6f01c7e4eb2db43321144c20313d90b735c8be553b00f7a73b303ca76"} Oct 31 00:20:14.376441 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.376380 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" podStartSLOduration=2.3763661320000002 podStartE2EDuration="2.376366132s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:14.373890663 +0000 UTC m=+1536.406986730" watchObservedRunningTime="2026-10-31 00:20:14.376366132 +0000 UTC m=+1536.409462202" Oct 31 00:20:14.386326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.386234 6234 scope.go:122] "RemoveContainer" containerID="72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897" Oct 31 00:20:14.386592 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:14.386566 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897\": container with ID starting with 72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897 not found: ID does not exist" containerID="72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897" Oct 31 00:20:14.386636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.386597 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897"} err="failed to get container status \"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897\": rpc error: code = NotFound desc = could not find container \"72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897\": container with ID starting with 72b68ba830f9f2e868ac43c7c0b64a892be34171989f20792930ff69b7bb0897 not found: ID does not exist" Oct 31 00:20:14.386636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.386616 6234 scope.go:122] "RemoveContainer" containerID="0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271" Oct 31 00:20:14.393770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.393743 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:20:14.397594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.397566 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2854448d-55478cc96f-2ngdl"] Oct 31 00:20:14.402832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.402811 6234 scope.go:122] "RemoveContainer" containerID="0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271" Oct 31 00:20:14.403518 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:14.403490 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271\": container with ID starting with 0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271 not found: ID does not exist" containerID="0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271" Oct 31 00:20:14.403688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.403659 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271"} err="failed to get container status \"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271\": rpc error: code = NotFound desc = could not find container \"0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271\": container with ID starting with 0aa1cec90259c583d46992848e4e248736eb0d4433d9db14dae575faaea8b271 not found: ID does not exist" Oct 31 00:20:14.416655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.416605 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" podStartSLOduration=2.416591411 podStartE2EDuration="2.416591411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:14.412141125 +0000 UTC m=+1536.445237216" watchObservedRunningTime="2026-10-31 00:20:14.416591411 +0000 UTC m=+1536.449687480" Oct 31 00:20:14.433616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.433584 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:20:14.438571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.438541 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2854448d-74ffb98875-jvjms"] Oct 31 00:20:14.569680 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.569654 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:20:14.619159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.619057 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\") pod \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " Oct 31 00:20:14.619324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.619181 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\") pod \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " Oct 31 00:20:14.619324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.619277 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\" (UniqueName: \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\") pod \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " Oct 31 00:20:14.619462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.619439 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\") pod \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\" (UID: \"e2e2000d-a2e5-4dd1-a102-7249f1d31970\") " Oct 31 00:20:14.619756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.619727 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca" pod "e2e2000d-a2e5-4dd1-a102-7249f1d31970" (UID: "e2e2000d-a2e5-4dd1-a102-7249f1d31970"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:14.620255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.620230 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2e2000d-a2e5-4dd1-a102-7249f1d31970-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.622390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.622365 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc" pod "e2e2000d-a2e5-4dd1-a102-7249f1d31970" (UID: "e2e2000d-a2e5-4dd1-a102-7249f1d31970"). InnerVolumeSpecName "kube-api-access-26lrc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:14.622530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.622446 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls" pod "e2e2000d-a2e5-4dd1-a102-7249f1d31970" (UID: "e2e2000d-a2e5-4dd1-a102-7249f1d31970"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:14.623103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.623077 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs" pod "e2e2000d-a2e5-4dd1-a102-7249f1d31970" (UID: "e2e2000d-a2e5-4dd1-a102-7249f1d31970"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:14.805810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.777656 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26lrc\" (UniqueName: \"kubernetes.io/projected/e2e2000d-a2e5-4dd1-a102-7249f1d31970-kube-api-access-26lrc\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.805810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.777700 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:14.805810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:14.777717 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e2e2000d-a2e5-4dd1-a102-7249f1d31970-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:15.362532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.362501 6234 generic.go:356] "Generic (PLEG): container finished" podID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" containerID="4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc" exitCode=0 Oct 31 00:20:15.363014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.362546 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" Oct 31 00:20:15.363014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.362586 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" event={"ID":"e2e2000d-a2e5-4dd1-a102-7249f1d31970","Type":"ContainerDied","Data":"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc"} Oct 31 00:20:15.363014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.362642 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np" event={"ID":"e2e2000d-a2e5-4dd1-a102-7249f1d31970","Type":"ContainerDied","Data":"9b202d6467d51a3f4d53aa60e3349c72969753b5d131ae9347a347eb47474ae5"} Oct 31 00:20:15.363014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.362667 6234 scope.go:122] "RemoveContainer" containerID="4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc" Oct 31 00:20:15.373675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.373656 6234 scope.go:122] "RemoveContainer" containerID="4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc" Oct 31 00:20:15.374006 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:15.373981 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc\": container with ID starting with 4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc not found: ID does not exist" containerID="4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc" Oct 31 00:20:15.374064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.374015 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc"} err="failed to get container status \"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc\": rpc error: code = NotFound desc = could not find container \"4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc\": container with ID starting with 4d9f2c8127a3d16e695dcaf62884d72a50f3f292a8d715d9e3597af8d05447fc not found: ID does not exist" Oct 31 00:20:15.396763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.396725 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:20:15.400375 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:15.400352 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-2854448d-7c77cd545d-vh2np"] Oct 31 00:20:16.084605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:16.084569 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" path="/var/lib/kubelet/pods/80b9c727-469f-4b4d-8fa5-c3a732f8145d/volumes" Oct 31 00:20:16.085107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:16.085091 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" path="/var/lib/kubelet/pods/e006a0ec-3727-4cd2-bd24-be3ca19ae0b8/volumes" Oct 31 00:20:16.085517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:16.085504 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" path="/var/lib/kubelet/pods/e2e2000d-a2e5-4dd1-a102-7249f1d31970/volumes" Oct 31 00:20:16.244626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:16.244584 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:20:16.244921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:16.244890 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" containerID="cri-o://3c27689f3d3a576b7810460c4dedfd3cb3ab6a1680398132e7a4f1933dd8dbf5" gracePeriod=30 Oct 31 00:20:21.057591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.057514 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:21.058355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058338 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" containerName="payload-processing" Oct 31 00:20:21.058402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058358 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" containerName="payload-processing" Oct 31 00:20:21.058402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058383 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" containerName="maas-api" Oct 31 00:20:21.058402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058388 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" containerName="maas-api" Oct 31 00:20:21.058494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058412 6234 cpu_manager.go:414] "containerMap: removing container" podUID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" containerName="payload-pre-processing" Oct 31 00:20:21.058494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058419 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" containerName="payload-pre-processing" Oct 31 00:20:21.058557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058532 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e006a0ec-3727-4cd2-bd24-be3ca19ae0b8" containerName="payload-processing" Oct 31 00:20:21.058557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058544 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2e2000d-a2e5-4dd1-a102-7249f1d31970" containerName="maas-api" Oct 31 00:20:21.058557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.058554 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="80b9c727-469f-4b4d-8fa5-c3a732f8145d" containerName="payload-pre-processing" Oct 31 00:20:21.063927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.063902 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.073384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.072907 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-15bcc310-serving-cert\"" Oct 31 00:20:21.076983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.076957 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:21.173585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.173551 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ntv8m\" (UniqueName: \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.173758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.173686 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.173931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.173912 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.173973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.173941 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.300867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.299627 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ntv8m\" (UniqueName: \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.300867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.299927 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.309739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.308961 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.309739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.309214 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.330887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.328526 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.353822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.345855 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.362691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.362601 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ntv8m\" (UniqueName: \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.367862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.367758 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\") pod \"maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.394374 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.394101 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:21.556324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.556289 6234 generic.go:356] "Generic (PLEG): container finished" podID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerID="3c27689f3d3a576b7810460c4dedfd3cb3ab6a1680398132e7a4f1933dd8dbf5" exitCode=0 Oct 31 00:20:21.556530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.556472 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerDied","Data":"3c27689f3d3a576b7810460c4dedfd3cb3ab6a1680398132e7a4f1933dd8dbf5"} Oct 31 00:20:21.556594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.556532 6234 scope.go:122] "RemoveContainer" containerID="977b78001590d50f6a14dd857fc283b7e5cdb0db046503e124c38322a918cb8f" Oct 31 00:20:21.752808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.752638 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:21.838827 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:21.828868 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba0de505_8416_4c31_9087_1154f3b555b7.slice/crio-0bd3e64c75ad0518ebd12047c3c3124346a659031df9953ae83b7b511ee54df8 WatchSource:0}: Error finding container 0bd3e64c75ad0518ebd12047c3c3124346a659031df9953ae83b7b511ee54df8: Status 404 returned error can't find the container with id 0bd3e64c75ad0518ebd12047c3c3124346a659031df9953ae83b7b511ee54df8 Oct 31 00:20:21.869686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:21.868551 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:20:22.162449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.162422 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:22.238733 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.238701 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.238948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.238743 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.238948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.238801 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.238948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.238919 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.239161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.238965 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.239161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239124 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.239161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239133 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:22.239318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239169 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:22.239318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239177 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.239318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239265 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.239472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.239365 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\") pod \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\" (UID: \"ec4925ae-5ad5-48f0-9a03-526b8fe2f573\") " Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240018 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240151 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240203 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240464 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240482 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240496 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240509 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.240549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.240524 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.242179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.242157 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:22.242309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.242293 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:20:22.243176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.243151 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "kube-api-access-9wnwk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:22.243262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.243201 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy" pod "ec4925ae-5ad5-48f0-9a03-526b8fe2f573" (UID: "ec4925ae-5ad5-48f0-9a03-526b8fe2f573"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:22.342287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.342254 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.342287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.342285 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9wnwk\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-kube-api-access-9wnwk\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.342487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.342301 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.342487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.342313 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ec4925ae-5ad5-48f0-9a03-526b8fe2f573-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:22.566146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.566110 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" event={"ID":"ba0de505-8416-4c31-9087-1154f3b555b7","Type":"ContainerStarted","Data":"0bd3e64c75ad0518ebd12047c3c3124346a659031df9953ae83b7b511ee54df8"} Oct 31 00:20:22.567547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.567525 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" event={"ID":"ec4925ae-5ad5-48f0-9a03-526b8fe2f573","Type":"ContainerDied","Data":"81e2499c3a71109b1c4c88691d375064a49f9bcea7f1d8952df8c262e4343173"} Oct 31 00:20:22.567631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.567547 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk" Oct 31 00:20:22.567631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.567576 6234 scope.go:122] "RemoveContainer" containerID="3c27689f3d3a576b7810460c4dedfd3cb3ab6a1680398132e7a4f1933dd8dbf5" Oct 31 00:20:22.601315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.601225 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:20:22.605998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:22.605976 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-2854448d-openshift-default-6f659658d7-wt9wk"] Oct 31 00:20:23.350021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:23.349990 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:23.353773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:23.353744 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:23.577060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:23.577021 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" event={"ID":"ba0de505-8416-4c31-9087-1154f3b555b7","Type":"ContainerStarted","Data":"f093dde6bf2d809688e459c628d5e4961d4ecb34b4ef92cfe9675a392561d0fa"} Oct 31 00:20:23.577243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:23.577086 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:23.604335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:23.604238 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" podStartSLOduration=2.012361775 podStartE2EDuration="2.604224551s" totalImagesPullingTime="591.862776ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:21 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:23.600442102 +0000 UTC m=+1545.633538174" watchObservedRunningTime="2026-10-31 00:20:23.604224551 +0000 UTC m=+1545.637320621" Oct 31 00:20:24.082983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:24.082941 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" path="/var/lib/kubelet/pods/ec4925ae-5ad5-48f0-9a03-526b8fe2f573/volumes" Oct 31 00:20:25.366128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.366101 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:25.366534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.366171 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:25.696729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.696693 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:20:25.697589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697571 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.697646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697592 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.697646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697609 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.697646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697614 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.697754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697744 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.697809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.697758 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ec4925ae-5ad5-48f0-9a03-526b8fe2f573" containerName="istio-proxy" Oct 31 00:20:25.703260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.703235 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.724120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.724086 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-1c9660-openshift-default-dockercfg-jks29\"" Oct 31 00:20:25.730016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.729982 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:20:25.816713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.816676 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.826810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.818158 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.827320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.827285 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rpwnc\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.827563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.827546 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.827734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.827718 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.828028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.828003 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.828236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.828212 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.828468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.828450 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.828667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.828651 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.935810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.935234 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.935810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.935631 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rpwnc\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.942837 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.943005 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.943586 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.943631 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.943697 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.944036 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.948858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.944104 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.961808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.955120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.961808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.955248 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.961808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.955371 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.961808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.956190 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.961808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.956417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.975899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.963681 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.975899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.965766 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rpwnc\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:25.995809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:25.983831 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:26.010851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.005514 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\") pod \"e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:26.027382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.026140 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:26.452693 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:26.452662 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd9a8230f_61fd_42ed_aa40_b9fc374be85e.slice/crio-f0d1efae3c48f3010aad52164ab3102ac2b35b7e52dca105f52c13d57be57483 WatchSource:0}: Error finding container f0d1efae3c48f3010aad52164ab3102ac2b35b7e52dca105f52c13d57be57483: Status 404 returned error can't find the container with id f0d1efae3c48f3010aad52164ab3102ac2b35b7e52dca105f52c13d57be57483 Oct 31 00:20:26.453039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.452801 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:20:26.455436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.455403 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:26.455519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.455468 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:26.455519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.455497 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:26.628059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:26.628029 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerStarted","Data":"f0d1efae3c48f3010aad52164ab3102ac2b35b7e52dca105f52c13d57be57483"} Oct 31 00:20:27.638017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:27.637972 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerStarted","Data":"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98"} Oct 31 00:20:27.672087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:27.672039 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podStartSLOduration=2.672024262 podStartE2EDuration="2.672024262s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:27.666759109 +0000 UTC m=+1549.699855181" watchObservedRunningTime="2026-10-31 00:20:27.672024262 +0000 UTC m=+1549.705120332" Oct 31 00:20:28.028288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:28.028254 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:28.034317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:28.034287 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:28.645697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:28.645666 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:28.646750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:28.646723 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:20:29.591166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:29.591132 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:31.594320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.594285 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:20:31.599040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.599019 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:31.611383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.611361 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-1c9660-dockercfg-zjxbp\"" Oct 31 00:20:31.612085 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.612068 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-1c9660\"" Oct 31 00:20:31.614071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.614052 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:20:31.639419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.639386 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:20:31.645554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.645522 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:31.660323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.660296 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:20:31.745810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.743335 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:31.745810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.743610 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:31.745810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.743744 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7xbx\" (UniqueName: \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:31.745810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.743826 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7dnh9\" (UniqueName: \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:31.949863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.932997 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7xbx\" (UniqueName: \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:31.949863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.933084 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7dnh9\" (UniqueName: \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:31.949863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.933229 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:31.949863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.933588 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:31.949863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:31.934615 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:32.032880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.019454 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:32.049229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.049193 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7xbx\" (UniqueName: \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\") pod \"payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:32.081665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.074375 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7dnh9\" (UniqueName: \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\") pod \"payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:32.217035 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.216943 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:32.276359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.276329 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:32.580171 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:32.580116 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73170cf3_9711_40a6_95a2_5e013a98bc12.slice/crio-e9b616cc9d1bea0d2d868418dc2cd2442a9d39ca5175b06b455c670291971d5f WatchSource:0}: Error finding container e9b616cc9d1bea0d2d868418dc2cd2442a9d39ca5175b06b455c670291971d5f: Status 404 returned error can't find the container with id e9b616cc9d1bea0d2d868418dc2cd2442a9d39ca5175b06b455c670291971d5f Oct 31 00:20:32.606834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.605771 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:20:32.628217 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:32.628187 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod374240e2_0c6a_4ec6_9db8_b07d279a49ad.slice/crio-7e2ede2d083a8bd2e68555bb74dcf6e596daac84e70509d1b4a8dbf31c57c0eb WatchSource:0}: Error finding container 7e2ede2d083a8bd2e68555bb74dcf6e596daac84e70509d1b4a8dbf31c57c0eb: Status 404 returned error can't find the container with id 7e2ede2d083a8bd2e68555bb74dcf6e596daac84e70509d1b4a8dbf31c57c0eb Oct 31 00:20:32.630219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.630110 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:20:32.701852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.701813 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" event={"ID":"374240e2-0c6a-4ec6-9db8-b07d279a49ad","Type":"ContainerStarted","Data":"7e2ede2d083a8bd2e68555bb74dcf6e596daac84e70509d1b4a8dbf31c57c0eb"} Oct 31 00:20:32.703681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:32.703658 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" event={"ID":"73170cf3-9711-40a6-95a2-5e013a98bc12","Type":"ContainerStarted","Data":"e9b616cc9d1bea0d2d868418dc2cd2442a9d39ca5175b06b455c670291971d5f"} Oct 31 00:20:33.163768 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:33.163727 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:20:33.713278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.713238 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" event={"ID":"374240e2-0c6a-4ec6-9db8-b07d279a49ad","Type":"ContainerStarted","Data":"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47"} Oct 31 00:20:33.713743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.713328 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:33.715160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.715132 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" event={"ID":"73170cf3-9711-40a6-95a2-5e013a98bc12","Type":"ContainerStarted","Data":"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb"} Oct 31 00:20:33.715273 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.715190 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:33.733185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.733123 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" podStartSLOduration=2.733104023 podStartE2EDuration="2.733104023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:33.732221258 +0000 UTC m=+1555.765317329" watchObservedRunningTime="2026-10-31 00:20:33.733104023 +0000 UTC m=+1555.766200094" Oct 31 00:20:33.755829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:33.755767 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" podStartSLOduration=2.755749852 podStartE2EDuration="2.755749852s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:33.752467736 +0000 UTC m=+1555.785563806" watchObservedRunningTime="2026-10-31 00:20:33.755749852 +0000 UTC m=+1555.788845923" Oct 31 00:20:44.726513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:44.726484 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:20:44.727010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:44.726559 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:20:45.061702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:45.061608 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:45.066661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:45.066610 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:46.519002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:46.518963 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:46.533359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:46.533322 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:50.062930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.062859 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" podUID="be069678-1488-40dc-afc9-30a761bf597d" containerName="payload-pre-processing" containerID="cri-o://0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293" gracePeriod=25 Oct 31 00:20:50.109459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.094423 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" podUID="5380d776-9b8f-42d1-99b7-ad7a1109541e" containerName="payload-processing" containerID="cri-o://d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e" gracePeriod=25 Oct 31 00:20:50.424264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.424225 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:50.478682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.478651 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:50.562900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.562867 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\") pod \"be069678-1488-40dc-afc9-30a761bf597d\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " Oct 31 00:20:50.563112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563089 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\" (UniqueName: \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\") pod \"be069678-1488-40dc-afc9-30a761bf597d\" (UID: \"be069678-1488-40dc-afc9-30a761bf597d\") " Oct 31 00:20:50.563185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563148 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\" (UniqueName: \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\") pod \"5380d776-9b8f-42d1-99b7-ad7a1109541e\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " Oct 31 00:20:50.563232 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563185 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\") pod \"5380d776-9b8f-42d1-99b7-ad7a1109541e\" (UID: \"5380d776-9b8f-42d1-99b7-ad7a1109541e\") " Oct 31 00:20:50.563370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563342 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume" pod "be069678-1488-40dc-afc9-30a761bf597d" (UID: "be069678-1488-40dc-afc9-30a761bf597d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:50.563669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563644 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume" pod "5380d776-9b8f-42d1-99b7-ad7a1109541e" (UID: "5380d776-9b8f-42d1-99b7-ad7a1109541e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:50.563955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563934 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/be069678-1488-40dc-afc9-30a761bf597d-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:50.563955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.563957 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5380d776-9b8f-42d1-99b7-ad7a1109541e-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:50.565465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.565445 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7" pod "5380d776-9b8f-42d1-99b7-ad7a1109541e" (UID: "5380d776-9b8f-42d1-99b7-ad7a1109541e"). InnerVolumeSpecName "kube-api-access-m57v7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:50.565532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.565476 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr" pod "be069678-1488-40dc-afc9-30a761bf597d" (UID: "be069678-1488-40dc-afc9-30a761bf597d"). InnerVolumeSpecName "kube-api-access-t2qwr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:50.666108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.666021 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t2qwr\" (UniqueName: \"kubernetes.io/projected/be069678-1488-40dc-afc9-30a761bf597d-kube-api-access-t2qwr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:50.666108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.666048 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m57v7\" (UniqueName: \"kubernetes.io/projected/5380d776-9b8f-42d1-99b7-ad7a1109541e-kube-api-access-m57v7\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:50.984211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.984181 6234 generic.go:356] "Generic (PLEG): container finished" podID="be069678-1488-40dc-afc9-30a761bf597d" containerID="0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293" exitCode=0 Oct 31 00:20:50.984401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.984229 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" Oct 31 00:20:50.984401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.984273 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" event={"ID":"be069678-1488-40dc-afc9-30a761bf597d","Type":"ContainerDied","Data":"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293"} Oct 31 00:20:50.984401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.984328 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv" event={"ID":"be069678-1488-40dc-afc9-30a761bf597d","Type":"ContainerDied","Data":"b79199a1b87b57bd248aa7c10f8d02e6c533abb4091f633c3bb814ad60367d2d"} Oct 31 00:20:50.984401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.984350 6234 scope.go:122] "RemoveContainer" containerID="0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293" Oct 31 00:20:50.986026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.986004 6234 generic.go:356] "Generic (PLEG): container finished" podID="5380d776-9b8f-42d1-99b7-ad7a1109541e" containerID="d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e" exitCode=0 Oct 31 00:20:50.986156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.986031 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" event={"ID":"5380d776-9b8f-42d1-99b7-ad7a1109541e","Type":"ContainerDied","Data":"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e"} Oct 31 00:20:50.986156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.986057 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" Oct 31 00:20:50.986156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.986082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f" event={"ID":"5380d776-9b8f-42d1-99b7-ad7a1109541e","Type":"ContainerDied","Data":"ef10d9a119e99ca31ad51fcaf6f05f9d6575b5e44f73f7282b6bf72877fd672c"} Oct 31 00:20:50.994931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.994914 6234 scope.go:122] "RemoveContainer" containerID="0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293" Oct 31 00:20:50.995233 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:50.995218 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293\": container with ID starting with 0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293 not found: ID does not exist" containerID="0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293" Oct 31 00:20:50.995305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.995244 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293"} err="failed to get container status \"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293\": rpc error: code = NotFound desc = could not find container \"0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293\": container with ID starting with 0734dd7552b6c4010c5299e50a016dd1a91183be53806ce147c417be36910293 not found: ID does not exist" Oct 31 00:20:50.995305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:50.995260 6234 scope.go:122] "RemoveContainer" containerID="d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e" Oct 31 00:20:51.005513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.005497 6234 scope.go:122] "RemoveContainer" containerID="d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e" Oct 31 00:20:51.005773 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:51.005750 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e\": container with ID starting with d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e not found: ID does not exist" containerID="d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e" Oct 31 00:20:51.005871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.005823 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e"} err="failed to get container status \"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e\": rpc error: code = NotFound desc = could not find container \"d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e\": container with ID starting with d1533519ec5858107f8296e97d75f2e09f0cf7e1a9013f4724f120982aa76e6e not found: ID does not exist" Oct 31 00:20:51.015506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.015485 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:51.020444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.020425 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-f71e7a13-5c9f7b4f67-mc98f"] Oct 31 00:20:51.033391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.033363 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:51.037853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.037826 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-f71e7a13-58c6474f9f-5kxdv"] Oct 31 00:20:51.522873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.521269 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" podUID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" containerName="payload-processing" containerID="cri-o://c9d87a8323905e3976ed257c9434dc35b5939810def60e0085a5c80033e0372f" gracePeriod=25 Oct 31 00:20:51.537418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:51.536905 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" podUID="a9a0989e-c1d2-401d-92cb-64380491919e" containerName="payload-pre-processing" containerID="cri-o://41229088e7b3d144a0ca4399de7780116405dac24d2931a581ebf44d73046cab" gracePeriod=25 Oct 31 00:20:52.001681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.001651 6234 generic.go:356] "Generic (PLEG): container finished" podID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" containerID="c9d87a8323905e3976ed257c9434dc35b5939810def60e0085a5c80033e0372f" exitCode=0 Oct 31 00:20:52.001956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.001714 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" event={"ID":"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8","Type":"ContainerDied","Data":"c9d87a8323905e3976ed257c9434dc35b5939810def60e0085a5c80033e0372f"} Oct 31 00:20:52.003504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.003480 6234 generic.go:356] "Generic (PLEG): container finished" podID="a9a0989e-c1d2-401d-92cb-64380491919e" containerID="41229088e7b3d144a0ca4399de7780116405dac24d2931a581ebf44d73046cab" exitCode=0 Oct 31 00:20:52.003633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.003518 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" event={"ID":"a9a0989e-c1d2-401d-92cb-64380491919e","Type":"ContainerDied","Data":"41229088e7b3d144a0ca4399de7780116405dac24d2931a581ebf44d73046cab"} Oct 31 00:20:52.081432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.081402 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:52.099872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.099837 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5380d776-9b8f-42d1-99b7-ad7a1109541e" path="/var/lib/kubelet/pods/5380d776-9b8f-42d1-99b7-ad7a1109541e/volumes" Oct 31 00:20:52.100533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.100504 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be069678-1488-40dc-afc9-30a761bf597d" path="/var/lib/kubelet/pods/be069678-1488-40dc-afc9-30a761bf597d/volumes" Oct 31 00:20:52.160660 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.160636 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:52.229266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.229234 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\" (UniqueName: \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\") pod \"a9a0989e-c1d2-401d-92cb-64380491919e\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " Oct 31 00:20:52.229451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.229430 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\" (UniqueName: \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\") pod \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " Oct 31 00:20:52.229522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.229507 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\") pod \"a9a0989e-c1d2-401d-92cb-64380491919e\" (UID: \"a9a0989e-c1d2-401d-92cb-64380491919e\") " Oct 31 00:20:52.229563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.229540 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\") pod \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\" (UID: \"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8\") " Oct 31 00:20:52.230702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.230674 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume" pod "a9a0989e-c1d2-401d-92cb-64380491919e" (UID: "a9a0989e-c1d2-401d-92cb-64380491919e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:52.230839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.230705 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume" pod "8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" (UID: "8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:52.233241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.233219 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4" pod "8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" (UID: "8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8"). InnerVolumeSpecName "kube-api-access-tsqj4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:52.233336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.233306 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n" pod "a9a0989e-c1d2-401d-92cb-64380491919e" (UID: "a9a0989e-c1d2-401d-92cb-64380491919e"). InnerVolumeSpecName "kube-api-access-qxn8n". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:52.331430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.331335 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a9a0989e-c1d2-401d-92cb-64380491919e-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:52.331430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.331380 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:52.331430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.331391 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxn8n\" (UniqueName: \"kubernetes.io/projected/a9a0989e-c1d2-401d-92cb-64380491919e-kube-api-access-qxn8n\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:52.331430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.331400 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsqj4\" (UniqueName: \"kubernetes.io/projected/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8-kube-api-access-tsqj4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:52.576578 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.576539 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:52.577093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.576800 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" podUID="ba0de505-8416-4c31-9087-1154f3b555b7" containerName="maas-api" containerID="cri-o://f093dde6bf2d809688e459c628d5e4961d4ecb34b4ef92cfe9675a392561d0fa" gracePeriod=30 Oct 31 00:20:53.091872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:52.905509 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:20:53.137035 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.136993 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" containerID="cri-o://d12190c4003252c4ca1a7e4ca68167f7b56cdf5701210804a71c8c3c8f74622f" gracePeriod=30 Oct 31 00:20:53.505867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.499491 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" event={"ID":"a9a0989e-c1d2-401d-92cb-64380491919e","Type":"ContainerDied","Data":"5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581"} Oct 31 00:20:53.505867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.499564 6234 scope.go:122] "RemoveContainer" containerID="41229088e7b3d144a0ca4399de7780116405dac24d2931a581ebf44d73046cab" Oct 31 00:20:53.505867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.499725 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w" Oct 31 00:20:53.538852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.538451 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" event={"ID":"8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8","Type":"ContainerDied","Data":"8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4"} Oct 31 00:20:53.538852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.538579 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22" Oct 31 00:20:53.588046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.587998 6234 generic.go:356] "Generic (PLEG): container finished" podID="ba0de505-8416-4c31-9087-1154f3b555b7" containerID="f093dde6bf2d809688e459c628d5e4961d4ecb34b4ef92cfe9675a392561d0fa" exitCode=0 Oct 31 00:20:53.588656 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.588631 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" event={"ID":"ba0de505-8416-4c31-9087-1154f3b555b7","Type":"ContainerDied","Data":"f093dde6bf2d809688e459c628d5e4961d4ecb34b4ef92cfe9675a392561d0fa"} Oct 31 00:20:53.602443 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.602411 6234 scope.go:122] "RemoveContainer" containerID="c9d87a8323905e3976ed257c9434dc35b5939810def60e0085a5c80033e0372f" Oct 31 00:20:53.900548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.900508 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:53.900888 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.900864 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:53.918870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.918836 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-15bcc310-5db8f997d7-wfx22"] Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.966422 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.968178 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\") pod \"ba0de505-8416-4c31-9087-1154f3b555b7\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.968432 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\") pod \"ba0de505-8416-4c31-9087-1154f3b555b7\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.968539 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\" (UniqueName: \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\") pod \"ba0de505-8416-4c31-9087-1154f3b555b7\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.968633 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\") pod \"ba0de505-8416-4c31-9087-1154f3b555b7\" (UID: \"ba0de505-8416-4c31-9087-1154f3b555b7\") " Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.971809 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca" pod "ba0de505-8416-4c31-9087-1154f3b555b7" (UID: "ba0de505-8416-4c31-9087-1154f3b555b7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:53.976296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.973248 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ba0de505-8416-4c31-9087-1154f3b555b7-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:53.990175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.990120 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs" pod "ba0de505-8416-4c31-9087-1154f3b555b7" (UID: "ba0de505-8416-4c31-9087-1154f3b555b7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:53.993314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.993264 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m" pod "ba0de505-8416-4c31-9087-1154f3b555b7" (UID: "ba0de505-8416-4c31-9087-1154f3b555b7"). InnerVolumeSpecName "kube-api-access-ntv8m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:53.999936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:53.999871 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-15bcc310-6458b48c55-p9j8w"] Oct 31 00:20:54.000879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.000837 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls" pod "ba0de505-8416-4c31-9087-1154f3b555b7" (UID: "ba0de505-8416-4c31-9087-1154f3b555b7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:20:54.034904 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:20:54.034648 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c947ed8_29bc_4a1a_8cd2_c0f15f1ba2c8.slice/crio-8321ae16fdaa13e908e5bb3c7f05d5c1e4b4258bf0beae86dc1c6afc90b4b4d4\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda9a0989e_c1d2_401d_92cb_64380491919e.slice/crio-5f237d20762fd0440ee4bdc64c7417994f5cfc8b34af1df0930b53ea8e9da581\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c947ed8_29bc_4a1a_8cd2_c0f15f1ba2c8.slice\": RecentStats: unable to find data in memory cache]" Oct 31 00:20:54.100400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.095652 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:54.100400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.095691 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ntv8m\" (UniqueName: \"kubernetes.io/projected/ba0de505-8416-4c31-9087-1154f3b555b7-kube-api-access-ntv8m\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:54.100400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.095707 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ba0de505-8416-4c31-9087-1154f3b555b7-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:54.105226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.105169 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" path="/var/lib/kubelet/pods/8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8/volumes" Oct 31 00:20:54.106487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.106392 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a9a0989e-c1d2-401d-92cb-64380491919e" path="/var/lib/kubelet/pods/a9a0989e-c1d2-401d-92cb-64380491919e/volumes" Oct 31 00:20:54.219551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.219507 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:20:54.220446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220424 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5380d776-9b8f-42d1-99b7-ad7a1109541e" containerName="payload-processing" Oct 31 00:20:54.220446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220445 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5380d776-9b8f-42d1-99b7-ad7a1109541e" containerName="payload-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220473 6234 cpu_manager.go:414] "containerMap: removing container" podUID="be069678-1488-40dc-afc9-30a761bf597d" containerName="payload-pre-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220480 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be069678-1488-40dc-afc9-30a761bf597d" containerName="payload-pre-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220514 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ba0de505-8416-4c31-9087-1154f3b555b7" containerName="maas-api" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220519 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ba0de505-8416-4c31-9087-1154f3b555b7" containerName="maas-api" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220527 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" containerName="payload-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220531 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" containerName="payload-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220539 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a9a0989e-c1d2-401d-92cb-64380491919e" containerName="payload-pre-processing" Oct 31 00:20:54.220576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220543 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a9a0989e-c1d2-401d-92cb-64380491919e" containerName="payload-pre-processing" Oct 31 00:20:54.220896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220642 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ba0de505-8416-4c31-9087-1154f3b555b7" containerName="maas-api" Oct 31 00:20:54.220896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220651 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c947ed8-29bc-4a1a-8cd2-c0f15f1ba2c8" containerName="payload-processing" Oct 31 00:20:54.220896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220657 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a9a0989e-c1d2-401d-92cb-64380491919e" containerName="payload-pre-processing" Oct 31 00:20:54.220896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220666 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5380d776-9b8f-42d1-99b7-ad7a1109541e" containerName="payload-processing" Oct 31 00:20:54.220896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.220675 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="be069678-1488-40dc-afc9-30a761bf597d" containerName="payload-pre-processing" Oct 31 00:20:54.226140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.226120 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.232444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.232418 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w1-0df3f8-openshift-default-dockercfg-dw7vf\"" Oct 31 00:20:54.251388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.251356 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:20:54.299378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299339 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.299597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299419 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.299669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299600 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.299760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299742 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2gftr\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.301400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299828 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.301400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.299874 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.301400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.300037 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.301400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.300085 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.301400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.300139 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.414666 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.414745 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.414824 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.414885 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.414964 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.415059 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.415242 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2gftr\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.415302 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.415363 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.415490 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.416961 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.417191 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.419037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.417711 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.425521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.425485 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.467817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.434375 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.471109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.471032 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.482440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.474099 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.489809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.483991 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2gftr\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.556126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.555836 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:54.646982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.646945 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" event={"ID":"ba0de505-8416-4c31-9087-1154f3b555b7","Type":"ContainerDied","Data":"0bd3e64c75ad0518ebd12047c3c3124346a659031df9953ae83b7b511ee54df8"} Oct 31 00:20:54.647518 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.647501 6234 scope.go:122] "RemoveContainer" containerID="f093dde6bf2d809688e459c628d5e4961d4ecb34b4ef92cfe9675a392561d0fa" Oct 31 00:20:54.647763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.647748 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l" Oct 31 00:20:54.744110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.744066 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:54.749849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.749816 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-15bcc310-5b6bcb45f8-c8k5l"] Oct 31 00:20:54.902882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.902847 6234 patch_prober.go:29] interesting pod/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:20:54.903053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:54.902926 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:20:55.039195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.039163 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:20:55.039550 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:55.039527 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod277e0d55_a194_44cf_a75f_771a24a9fb65.slice/crio-88ba7b119c9a8790a5aed1d0895f0021a6f3579b79dcab014d7bff34eb31366e WatchSource:0}: Error finding container 88ba7b119c9a8790a5aed1d0895f0021a6f3579b79dcab014d7bff34eb31366e: Status 404 returned error can't find the container with id 88ba7b119c9a8790a5aed1d0895f0021a6f3579b79dcab014d7bff34eb31366e Oct 31 00:20:55.042296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.042260 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:55.042766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.042341 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:55.042766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.042425 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:55.663910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.663018 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerStarted","Data":"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9"} Oct 31 00:20:55.663910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.663097 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerStarted","Data":"88ba7b119c9a8790a5aed1d0895f0021a6f3579b79dcab014d7bff34eb31366e"} Oct 31 00:20:55.695485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:55.695435 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podStartSLOduration=1.6954198150000002 podStartE2EDuration="1.695419815s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:55.691389098 +0000 UTC m=+1577.724485182" watchObservedRunningTime="2026-10-31 00:20:55.695419815 +0000 UTC m=+1577.728515884" Oct 31 00:20:56.083974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.083942 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ba0de505-8416-4c31-9087-1154f3b555b7" path="/var/lib/kubelet/pods/ba0de505-8416-4c31-9087-1154f3b555b7/volumes" Oct 31 00:20:56.221678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.221645 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:20:56.226756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.226722 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.231860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.231832 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-14e39b-openshift-default-dockercfg-rtll2\"" Oct 31 00:20:56.249913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.249876 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408409 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408469 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408519 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x99xw\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408607 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408677 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408716 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.408741 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.409081 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.414856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.409181 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514370 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514411 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514434 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514581 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514706 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514765 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514833 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.514893 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x99xw\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.518812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.515801 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.528804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.524526 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.537372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.537116 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.537528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.537377 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.545808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.545492 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.570816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.558992 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:56.570816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.560367 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.597813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.595905 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.614616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.614511 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x99xw\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.634382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.634314 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\") pod \"e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:56.719191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.719162 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:56.872675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:56.867118 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:57.267189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.267155 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:20:57.267368 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:20:57.267345 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01b77f4c_be6f_4bec_ad2c_90f342187369.slice/crio-4388d6952b3e97ff1b4e037f21254051a984997e3b18229b93030f95ec88605f WatchSource:0}: Error finding container 4388d6952b3e97ff1b4e037f21254051a984997e3b18229b93030f95ec88605f: Status 404 returned error can't find the container with id 4388d6952b3e97ff1b4e037f21254051a984997e3b18229b93030f95ec88605f Oct 31 00:20:57.270011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.269980 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:57.270075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.270042 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:57.270108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.270073 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:20:57.726436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.725498 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerStarted","Data":"3b11d0dfe4cf3fba1984d0b8cad201ca35d0cdcf9870e33fc14a8d40c2cd7a26"} Oct 31 00:20:57.726436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.725611 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerStarted","Data":"4388d6952b3e97ff1b4e037f21254051a984997e3b18229b93030f95ec88605f"} Oct 31 00:20:57.726436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.726034 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:57.727726 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.727704 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:20:57.758124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.758074 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podStartSLOduration=1.758052368 podStartE2EDuration="1.758052368s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:20:57.752897906 +0000 UTC m=+1579.785993978" watchObservedRunningTime="2026-10-31 00:20:57.758052368 +0000 UTC m=+1579.791148438" Oct 31 00:20:57.867537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:57.867507 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:58.736634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:58.736602 6234 generic.go:356] "Generic (PLEG): container finished" podID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerID="d12190c4003252c4ca1a7e4ca68167f7b56cdf5701210804a71c8c3c8f74622f" exitCode=0 Oct 31 00:20:58.737095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:58.736670 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" event={"ID":"38a9b733-9eae-40c9-bca2-77b56be7f749","Type":"ContainerDied","Data":"d12190c4003252c4ca1a7e4ca68167f7b56cdf5701210804a71c8c3c8f74622f"} Oct 31 00:20:58.874832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:58.874806 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:58.903678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:58.903652 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:59.042071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042036 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042113 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042160 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042291 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042355 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042495 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042400 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:20:59.042549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042504 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042573 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042581 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.042707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042610 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042611 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.042707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042686 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\") pod \"38a9b733-9eae-40c9-bca2-77b56be7f749\" (UID: \"38a9b733-9eae-40c9-bca2-77b56be7f749\") " Oct 31 00:20:59.042892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042806 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.042950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.042926 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.043892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.043865 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.043892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.043893 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.044067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.043908 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38a9b733-9eae-40c9-bca2-77b56be7f749-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.044067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.043923 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.044067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.043938 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.044567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.044543 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:20:59.044743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.044727 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:20:59.044855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.044771 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:59.045076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.045059 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm" pod "38a9b733-9eae-40c9-bca2-77b56be7f749" (UID: "38a9b733-9eae-40c9-bca2-77b56be7f749"). InnerVolumeSpecName "kube-api-access-mtpmm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:20:59.145748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.145660 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.145748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.145689 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mtpmm\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-kube-api-access-mtpmm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.145748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.145703 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.145748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.145716 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38a9b733-9eae-40c9-bca2-77b56be7f749-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:20:59.539934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.539901 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:20:59.540638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.540625 6234 cpu_manager.go:414] "containerMap: removing container" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" Oct 31 00:20:59.540693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.540639 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" Oct 31 00:20:59.540778 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.540769 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" containerName="istio-proxy" Oct 31 00:20:59.544756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.544739 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.555420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.555371 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w1-0df3f8\"" Oct 31 00:20:59.559134 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.559091 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w1-0df3f8-dockercfg-gqgrr\"" Oct 31 00:20:59.560294 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.560270 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:20:59.564310 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.564287 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:20:59.568814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.568792 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.585419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.585390 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:20:59.742881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.734352 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.742881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.734453 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-std5c\" (UniqueName: \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.742881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.734606 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hhhxw\" (UniqueName: \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.742881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.734750 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.847864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.844773 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hhhxw\" (UniqueName: \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.855808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.844938 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.855808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.855768 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.856056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.855881 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-std5c\" (UniqueName: \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.872958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.868415 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:20:59.878808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.876927 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.926501 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.929010 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hhhxw\" (UniqueName: \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\") pod \"payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.930033 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz" event={"ID":"38a9b733-9eae-40c9-bca2-77b56be7f749","Type":"ContainerDied","Data":"3206337f035de271f7331aaff45114f7e852d778dcec567e60d104e90ce514d5"} Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.930130 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.930165 6234 scope.go:122] "RemoveContainer" containerID="d12190c4003252c4ca1a7e4ca68167f7b56cdf5701210804a71c8c3c8f74622f" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.930836 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:20:59.934809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.932881 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:20:59.978814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:20:59.976910 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-std5c\" (UniqueName: \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\") pod \"payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:21:00.105061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.104697 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:21:00.105061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.104750 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-f71e7a13-gw-openshift-default-8c6d5dd4f-bsjsz"] Oct 31 00:21:00.162677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.162287 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:21:00.576027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.575724 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:21:00.581132 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:00.581072 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod08486c3c_312d_4813_af56_8b8a597b2a7a.slice/crio-82882309f1de1b645cb1ecc10fd04708cef87f11aa446863c9ef393d2a9f1ae7 WatchSource:0}: Error finding container 82882309f1de1b645cb1ecc10fd04708cef87f11aa446863c9ef393d2a9f1ae7: Status 404 returned error can't find the container with id 82882309f1de1b645cb1ecc10fd04708cef87f11aa446863c9ef393d2a9f1ae7 Oct 31 00:21:00.841867 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:00.841828 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode876ba6b_d1fe_497c_8714_bda90849eefd.slice/crio-aa920c2e5510a147ef416ba179360bebe573a802ff72c1e842f6931c26af258d WatchSource:0}: Error finding container aa920c2e5510a147ef416ba179360bebe573a802ff72c1e842f6931c26af258d: Status 404 returned error can't find the container with id aa920c2e5510a147ef416ba179360bebe573a802ff72c1e842f6931c26af258d Oct 31 00:21:00.842369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.842345 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:21:00.937005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.933932 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" event={"ID":"08486c3c-312d-4813-af56-8b8a597b2a7a","Type":"ContainerStarted","Data":"60216f290ff2765157db08166a33f2f35b01a413619e8f5095a35a8f05df0b2a"} Oct 31 00:21:00.937005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.933993 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" event={"ID":"08486c3c-312d-4813-af56-8b8a597b2a7a","Type":"ContainerStarted","Data":"82882309f1de1b645cb1ecc10fd04708cef87f11aa446863c9ef393d2a9f1ae7"} Oct 31 00:21:00.937005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.934028 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:21:00.937005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.936075 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" event={"ID":"e876ba6b-d1fe-497c-8714-bda90849eefd","Type":"ContainerStarted","Data":"aa920c2e5510a147ef416ba179360bebe573a802ff72c1e842f6931c26af258d"} Oct 31 00:21:00.998177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:00.998126 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" podStartSLOduration=1.9981125259999999 podStartE2EDuration="1.998112526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:00.996997332 +0000 UTC m=+1583.030093427" watchObservedRunningTime="2026-10-31 00:21:00.998112526 +0000 UTC m=+1583.031208596" Oct 31 00:21:01.503459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.503425 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:21:01.508094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.508074 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.517632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.517600 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-41a0b7\"" Oct 31 00:21:01.517965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.517949 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-41a0b7-dockercfg-9lvlg\"" Oct 31 00:21:01.520231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.520210 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:21:01.549370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.549331 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dt6lp\" (UniqueName: \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.549541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.549512 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.550732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.550707 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:21:01.555401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.555383 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:01.571195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.570918 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:21:01.734815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.708581 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.846954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.844321 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dt6lp\" (UniqueName: \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.846954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.728859 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.915805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.915555 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dt6lp\" (UniqueName: \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\") pod \"payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:01.958808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.958146 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:01.958808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:01.958274 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lv45m\" (UniqueName: \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.016810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.016299 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" event={"ID":"e876ba6b-d1fe-497c-8714-bda90849eefd","Type":"ContainerStarted","Data":"dc6cf0bd14a072c80a6eef8581211289a2485b05a02d5fa2495873ac676b9b42"} Oct 31 00:21:02.016810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.016360 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:21:02.059823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.059750 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.060018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.059882 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lv45m\" (UniqueName: \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.060847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.060768 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.064752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.064688 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" podStartSLOduration=3.064672489 podStartE2EDuration="3.064672489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:20:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:02.061154999 +0000 UTC m=+1584.094251072" watchObservedRunningTime="2026-10-31 00:21:02.064672489 +0000 UTC m=+1584.097768559" Oct 31 00:21:02.094661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.094618 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38a9b733-9eae-40c9-bca2-77b56be7f749" path="/var/lib/kubelet/pods/38a9b733-9eae-40c9-bca2-77b56be7f749/volumes" Oct 31 00:21:02.096671 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.096642 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lv45m\" (UniqueName: \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\") pod \"payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.124163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.124059 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:02.175288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.175245 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:02.507496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.507461 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:21:02.507709 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:02.507688 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9538ca46_c9b8_4fb6_9a75_ef8eb2e5a0e9.slice/crio-9d849522dc87418ac255448a8dc9d73b9a36f32f107b884a6ba536cb35223797 WatchSource:0}: Error finding container 9d849522dc87418ac255448a8dc9d73b9a36f32f107b884a6ba536cb35223797: Status 404 returned error can't find the container with id 9d849522dc87418ac255448a8dc9d73b9a36f32f107b884a6ba536cb35223797 Oct 31 00:21:02.643210 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:02.643177 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3486bbdd_f413_4abb_9f7a_50d56787a7f6.slice/crio-c75810fe4f8b622b48e729b3953429da741bd0d3f0e1a232d59914a1ddede594 WatchSource:0}: Error finding container c75810fe4f8b622b48e729b3953429da741bd0d3f0e1a232d59914a1ddede594: Status 404 returned error can't find the container with id c75810fe4f8b622b48e729b3953429da741bd0d3f0e1a232d59914a1ddede594 Oct 31 00:21:02.643210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:02.643199 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:21:03.142806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.123227 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" event={"ID":"3486bbdd-f413-4abb-9f7a-50d56787a7f6","Type":"ContainerStarted","Data":"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59"} Oct 31 00:21:03.142806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.123304 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" event={"ID":"3486bbdd-f413-4abb-9f7a-50d56787a7f6","Type":"ContainerStarted","Data":"c75810fe4f8b622b48e729b3953429da741bd0d3f0e1a232d59914a1ddede594"} Oct 31 00:21:03.142806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.124574 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:03.171859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.164921 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" event={"ID":"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9","Type":"ContainerStarted","Data":"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d"} Oct 31 00:21:03.171859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.164980 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:03.171859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.165001 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" event={"ID":"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9","Type":"ContainerStarted","Data":"9d849522dc87418ac255448a8dc9d73b9a36f32f107b884a6ba536cb35223797"} Oct 31 00:21:03.171859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.171568 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" podStartSLOduration=2.171548772 podStartE2EDuration="2.171548772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:03.169926041 +0000 UTC m=+1585.203022123" watchObservedRunningTime="2026-10-31 00:21:03.171548772 +0000 UTC m=+1585.204644845" Oct 31 00:21:03.199952 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.199901 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" podStartSLOduration=2.1998832520000002 podStartE2EDuration="2.199883252s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:03.196069433 +0000 UTC m=+1585.229165503" watchObservedRunningTime="2026-10-31 00:21:03.199883252 +0000 UTC m=+1585.232979321" Oct 31 00:21:03.218209 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:21:03.218174 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:21:03.559999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.559956 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:21:03.564960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.564935 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.574336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.573997 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-14e39b\"" Oct 31 00:21:03.579084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.579049 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:21:03.625057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.625019 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.625353 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.625331 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wb4dw\" (UniqueName: \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.732744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.731658 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.733001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.732861 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wb4dw\" (UniqueName: \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.733494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.732702 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.867571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.867487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wb4dw\" (UniqueName: \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\") pod \"payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:03.903815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:03.900152 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:04.529819 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:04.529765 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2baff5db_503a_4da9_875c_3e247765945f.slice/crio-66cbc2af48801c7b644d81fd3bf048fe36c4e54b4a2f4c7689b32cd9a6332bd4 WatchSource:0}: Error finding container 66cbc2af48801c7b644d81fd3bf048fe36c4e54b4a2f4c7689b32cd9a6332bd4: Status 404 returned error can't find the container with id 66cbc2af48801c7b644d81fd3bf048fe36c4e54b4a2f4c7689b32cd9a6332bd4 Oct 31 00:21:04.530257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:04.530237 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:21:05.211900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:05.211859 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" event={"ID":"2baff5db-503a-4da9-875c-3e247765945f","Type":"ContainerStarted","Data":"5951c8a43884faa3efd40312b86bc71a0d70cc10de929729592d869c5c3c61f4"} Oct 31 00:21:05.212068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:05.211921 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" event={"ID":"2baff5db-503a-4da9-875c-3e247765945f","Type":"ContainerStarted","Data":"66cbc2af48801c7b644d81fd3bf048fe36c4e54b4a2f4c7689b32cd9a6332bd4"} Oct 31 00:21:05.212638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:05.212619 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:05.238552 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:05.238499 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" podStartSLOduration=2.238483038 podStartE2EDuration="2.238483038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:05.23351647 +0000 UTC m=+1587.266612541" watchObservedRunningTime="2026-10-31 00:21:05.238483038 +0000 UTC m=+1587.271579151" Oct 31 00:21:12.020654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:12.020624 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:21:13.168820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:13.168766 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:21:13.978030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:13.977993 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:21:13.984265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:13.984238 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:13.995158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:13.991557 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w2-14e39b-serving-cert\"" Oct 31 00:21:14.003633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.003584 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:21:14.038467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.038428 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.038688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.038585 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.038688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.038660 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9rnkr\" (UniqueName: \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.038913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.038862 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.154826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.146666 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9rnkr\" (UniqueName: \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.154826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.146926 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.154826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.147155 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.154826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.147317 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.154826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.148915 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.172014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.171810 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9rnkr\" (UniqueName: \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.226879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.213553 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.233889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.233292 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\") pod \"maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.248668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.242447 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:21:14.309297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.309255 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:14.694959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.694912 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:21:14.695213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.694990 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:14.715389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:14.715357 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:21:14.716931 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:14.716903 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod47eb3460_dd38_4319_ba98_5ddf0c9011cf.slice/crio-ac89cc3003def497c3ea6dd0d2a089a26f3ff2a68612d693585da10d752019dd WatchSource:0}: Error finding container ac89cc3003def497c3ea6dd0d2a089a26f3ff2a68612d693585da10d752019dd: Status 404 returned error can't find the container with id ac89cc3003def497c3ea6dd0d2a089a26f3ff2a68612d693585da10d752019dd Oct 31 00:21:15.216736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:15.216631 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:21:15.484196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:15.484162 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" event={"ID":"47eb3460-dd38-4319-ba98-5ddf0c9011cf","Type":"ContainerStarted","Data":"9cc5793cb30d5b34cafa7a3297140a92107fca2c13ddde391c4bfb1015dcc14c"} Oct 31 00:21:15.484196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:15.484228 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" event={"ID":"47eb3460-dd38-4319-ba98-5ddf0c9011cf","Type":"ContainerStarted","Data":"ac89cc3003def497c3ea6dd0d2a089a26f3ff2a68612d693585da10d752019dd"} Oct 31 00:21:15.484539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:15.484381 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:15.516319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:15.516254 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" podStartSLOduration=2.057478249 podStartE2EDuration="2.516233625s" totalImagesPullingTime="458.755376ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:15.513106837 +0000 UTC m=+1597.546202907" watchObservedRunningTime="2026-10-31 00:21:15.516233625 +0000 UTC m=+1597.549329698" Oct 31 00:21:16.259242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.259206 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:21:16.494566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.494536 6234 generic.go:356] "Generic (PLEG): container finished" podID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerID="5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98" exitCode=0 Oct 31 00:21:16.494764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.494611 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerDied","Data":"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98"} Oct 31 00:21:16.495398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.495376 6234 scope.go:122] "RemoveContainer" containerID="5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98" Oct 31 00:21:16.496961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.496934 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:16.497030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.496998 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:16.497067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:16.497029 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:17.505178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:17.505144 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerStarted","Data":"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061"} Oct 31 00:21:18.028511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:18.028478 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:21:18.028742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:18.028721 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:18.028801 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:18.028769 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:19.029128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:19.029086 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:19.029624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:19.029140 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:20.028307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:20.028266 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:20.028516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:20.028339 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:21.028495 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:21.028454 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:21.028972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:21.028527 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:21.521388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:21.519981 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:21:22.028354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:22.028320 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:22.028565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:22.028381 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:23.029152 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:23.029112 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" start-of-body= Oct 31 00:21:23.029665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:23.029182 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.103:15021/healthz/ready\": dial tcp 10.134.0.103:15021: connect: connection refused" Oct 31 00:21:24.032564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:24.032525 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:21:24.033062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:24.033038 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:21:24.033871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:24.033849 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:21:29.206015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.205949 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.389354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.389308 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:21:29.389708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.389675 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.671161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.670883 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:21:29.676820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.672344 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:21:29.715811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.688145 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.762095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.761361 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.799135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.799089 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.799964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.799933 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.831810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.821354 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:21:29.831810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.821440 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.882214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.882168 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.893816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.891037 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:29.929867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:29.919877 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.207695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.207065 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:21:30.397815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.391216 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.538876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.487477 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:21:30.680183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.679926 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:21:30.692005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.691956 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:21:30.760352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.731940 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.760352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.748550 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:21:30.776811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.772228 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.776811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.772691 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:21:30.800811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.800692 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.811326 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.870259 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.870331 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.811363 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.871971 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.888154 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.897905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.889416 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:21:30.901969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.900972 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:30.901969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:30.901705 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:21:31.338972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.338925 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:31.365257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.350917 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:21:31.429007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.428959 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:31.732959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.732759 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:21:31.734837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.734816 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:21:31.735109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.735096 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:21:31.805653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.805620 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:21:31.884983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:31.884195 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:21:32.036857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.036727 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:21:32.055815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.039739 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": context deadline exceeded" Oct 31 00:21:32.116722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.104222 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:21:32.513948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.511815 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:21:32.513948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.511893 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:21:32.513948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.511929 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:21:32.513948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:32.511969 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:21:33.190373 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:21:33.190331 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:21:33.399189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.399152 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:21:33.406507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.406477 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.418832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.418734 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:21:33.420720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.420692 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-08673d2c-openshift-default-dockercfg-mrvl7\"" Oct 31 00:21:33.593275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.593188 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.613860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.613733 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.620196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.620157 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.626356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.626305 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.653796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.653732 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.654269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.654245 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4qrs7\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.654670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.654649 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.655023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.655003 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.655493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.655472 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759272 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759337 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759481 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759592 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759652 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4qrs7\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759721 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759751 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.760494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.759891 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.773827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.765374 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.773827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.765592 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.773827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.765812 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.773827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.766058 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.773827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.766287 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.809843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.802940 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.819244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.814270 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.819244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.818367 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:33.840147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:33.834748 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4qrs7\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\") pod \"e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:34.048753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:34.048460 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:34.420015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:34.419987 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:21:34.420690 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:34.420648 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb0d1e8b_f5dc_48a1_889e_f15d91e9eb88.slice/crio-d4bb8642575343739424e4ff8c4db7fff44e598a51c8058995dbbeeda23bdcdd WatchSource:0}: Error finding container d4bb8642575343739424e4ff8c4db7fff44e598a51c8058995dbbeeda23bdcdd: Status 404 returned error can't find the container with id d4bb8642575343739424e4ff8c4db7fff44e598a51c8058995dbbeeda23bdcdd Oct 31 00:21:34.423746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:34.423711 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:34.423851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:34.423815 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:34.423851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:34.423845 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:35.101327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.101285 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerStarted","Data":"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363"} Oct 31 00:21:35.101745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.101392 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerStarted","Data":"d4bb8642575343739424e4ff8c4db7fff44e598a51c8058995dbbeeda23bdcdd"} Oct 31 00:21:35.156391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.156335 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podStartSLOduration=2.156318013 podStartE2EDuration="2.156318013s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:35.148700228 +0000 UTC m=+1617.181796299" watchObservedRunningTime="2026-10-31 00:21:35.156318013 +0000 UTC m=+1617.189414082" Oct 31 00:21:35.956900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.956871 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:21:35.962732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.962699 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:35.976103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.976077 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w4-a39be4\"" Oct 31 00:21:35.979603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:35.979578 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:21:36.049764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.049725 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:36.055721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.055694 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:36.063660 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.062218 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8l2nc\" (UniqueName: \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.063660 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.062457 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.190809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.185510 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8l2nc\" (UniqueName: \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.190809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.185863 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.248910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.222152 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.312808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.286835 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:36.312808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.299908 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8l2nc\" (UniqueName: \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\") pod \"payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.322809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.321172 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:21:36.597919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.592443 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:36.998626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:36.998595 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:21:36.999752 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:36.999721 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9546dbe9_0f87_4b6a_bc1a_3f5440c4b629.slice/crio-eb7577998c8bfe28537901269adb0fed1befb6f9bd3d20e13f3187ab957b3150 WatchSource:0}: Error finding container eb7577998c8bfe28537901269adb0fed1befb6f9bd3d20e13f3187ab957b3150: Status 404 returned error can't find the container with id eb7577998c8bfe28537901269adb0fed1befb6f9bd3d20e13f3187ab957b3150 Oct 31 00:21:37.329373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:37.326894 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" event={"ID":"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629","Type":"ContainerStarted","Data":"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424"} Oct 31 00:21:37.329373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:37.326977 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" event={"ID":"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629","Type":"ContainerStarted","Data":"eb7577998c8bfe28537901269adb0fed1befb6f9bd3d20e13f3187ab957b3150"} Oct 31 00:21:37.329373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:37.327030 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:37.372498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:37.372154 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" podStartSLOduration=2.3721330050000002 podStartE2EDuration="2.372133005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:37.362282556 +0000 UTC m=+1619.395378630" watchObservedRunningTime="2026-10-31 00:21:37.372133005 +0000 UTC m=+1619.405229075" Oct 31 00:21:39.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.223543 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:21:39.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.262880 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.266842 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:21:39.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.279567 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08673d2c\"" Oct 31 00:21:39.291842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.279709 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08673d2c-dockercfg-hx9b6\"" Oct 31 00:21:39.472288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.472249 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qq52p\" (UniqueName: \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.472669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.472647 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.536132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.527029 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:21:39.628291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.628196 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.628885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.628860 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qq52p\" (UniqueName: \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.629937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.629914 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.641065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.641041 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.658289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.651484 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:21:39.658289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.653763 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qq52p\" (UniqueName: \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\") pod \"payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.676126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.676101 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:39.834974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.834939 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.835231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.835206 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxzt8\" (UniqueName: \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.942528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.942478 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxzt8\" (UniqueName: \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.942768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.942748 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.943962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.943933 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:39.957603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:39.957574 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxzt8\" (UniqueName: \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:40.003620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.003588 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:40.014050 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:40.014018 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podebb30701_65ba_4746_bd1f_adc2f4a8acd6.slice/crio-b304850300b74ca0d38fe462a58f8c93e3f77c203711d73dcd03d810bab9b4ab WatchSource:0}: Error finding container b304850300b74ca0d38fe462a58f8c93e3f77c203711d73dcd03d810bab9b4ab: Status 404 returned error can't find the container with id b304850300b74ca0d38fe462a58f8c93e3f77c203711d73dcd03d810bab9b4ab Oct 31 00:21:40.014155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.014071 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:21:40.314511 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:40.314467 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8ac39d7_28c4_48eb_a612_506a2d1df18d.slice/crio-8fc09ed24d18f748cb6c9cc68c2d1401e30fe95e295fa191b8251904e57e8305 WatchSource:0}: Error finding container 8fc09ed24d18f748cb6c9cc68c2d1401e30fe95e295fa191b8251904e57e8305: Status 404 returned error can't find the container with id 8fc09ed24d18f748cb6c9cc68c2d1401e30fe95e295fa191b8251904e57e8305 Oct 31 00:21:40.315459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.315397 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:21:40.569565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.569472 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" event={"ID":"a8ac39d7-28c4-48eb-a612-506a2d1df18d","Type":"ContainerStarted","Data":"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381"} Oct 31 00:21:40.569565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.569527 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" event={"ID":"a8ac39d7-28c4-48eb-a612-506a2d1df18d","Type":"ContainerStarted","Data":"8fc09ed24d18f748cb6c9cc68c2d1401e30fe95e295fa191b8251904e57e8305"} Oct 31 00:21:40.569766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.569583 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:40.571403 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.571381 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" event={"ID":"ebb30701-65ba-4746-bd1f-adc2f4a8acd6","Type":"ContainerStarted","Data":"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c"} Oct 31 00:21:40.571521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.571422 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" event={"ID":"ebb30701-65ba-4746-bd1f-adc2f4a8acd6","Type":"ContainerStarted","Data":"b304850300b74ca0d38fe462a58f8c93e3f77c203711d73dcd03d810bab9b4ab"} Oct 31 00:21:40.571521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.571464 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:40.573091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.573072 6234 generic.go:356] "Generic (PLEG): container finished" podID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerID="48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9" exitCode=0 Oct 31 00:21:40.573170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.573114 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerDied","Data":"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9"} Oct 31 00:21:40.573430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.573417 6234 scope.go:122] "RemoveContainer" containerID="48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9" Oct 31 00:21:40.575102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.575076 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:40.575170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.575142 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:40.575214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.575176 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:21:40.593420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.593384 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" podStartSLOduration=1.5933712309999999 podStartE2EDuration="1.593371231s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:40.589154083 +0000 UTC m=+1622.622250148" watchObservedRunningTime="2026-10-31 00:21:40.593371231 +0000 UTC m=+1622.626467301" Oct 31 00:21:40.611841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:40.611800 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" podStartSLOduration=1.611775036 podStartE2EDuration="1.611775036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:40.609191196 +0000 UTC m=+1622.642287266" watchObservedRunningTime="2026-10-31 00:21:40.611775036 +0000 UTC m=+1622.644871105" Oct 31 00:21:41.724449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:41.724406 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerStarted","Data":"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176"} Oct 31 00:21:42.562052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:42.562017 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:21:42.562324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:42.562292 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:21:42.564122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:42.562485 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:21:42.743257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:42.743214 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:21:43.560809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.559218 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:21:43.560809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.559280 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:21:43.623744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.623708 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:21:43.629072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.629048 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.634305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.634280 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w1-7a7688-serving-cert\"" Oct 31 00:21:43.671807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.639292 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:21:43.679843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.679757 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.679843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.679841 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.680047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.680005 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.680177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.680157 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2sw57\" (UniqueName: \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.790428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.789181 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.790428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.789398 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2sw57\" (UniqueName: \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.790428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.789549 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.790428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.789623 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.792015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.791689 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.794322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.794232 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.795154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.795132 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.805285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.805158 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2sw57\" (UniqueName: \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\") pod \"maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:43.991126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:43.991090 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:44.386053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:44.386024 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:21:44.633879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:44.632120 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:21:44.633879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:44.632189 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:21:44.838221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:44.838183 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" event={"ID":"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1","Type":"ContainerStarted","Data":"439c40c22bc6eb3c3bb0f41822854c2d1ffd37b177b88c506531878349e0f965"} Oct 31 00:21:45.563948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:45.562438 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:21:45.563948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:45.562506 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:21:45.869183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:45.869085 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" event={"ID":"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1","Type":"ContainerStarted","Data":"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e"} Oct 31 00:21:45.869183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:45.869169 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:45.894496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:45.894441 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" podStartSLOduration=2.273853085 podStartE2EDuration="2.894425525s" totalImagesPullingTime="620.57244ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:43 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:45.890940559 +0000 UTC m=+1627.924036630" watchObservedRunningTime="2026-10-31 00:21:45.894425525 +0000 UTC m=+1627.927521594" Oct 31 00:21:46.561810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:46.561250 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" start-of-body= Oct 31 00:21:46.561810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:46.561318 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.106:15021/healthz/ready\": dial tcp 10.134.0.106:15021: connect: connection refused" Oct 31 00:21:47.563095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:47.563017 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:21:47.564308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:47.564277 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:21:48.327840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:48.327764 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:21:49.042335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.042298 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:21:49.049701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.049673 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.068720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.068689 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-a39be4-serving-cert\"" Oct 31 00:21:49.076385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.076358 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:21:49.248887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.248850 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7tfpp\" (UniqueName: \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.249841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.249802 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.250803 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.250760 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.251097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.251074 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.362714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.361691 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.362714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.361885 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.362714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.362316 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7tfpp\" (UniqueName: \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.362714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.362395 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.365756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.365731 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.377177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.376610 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.378181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.378157 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.379979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.379937 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7tfpp\" (UniqueName: \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\") pod \"maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:49.671106 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:49.671023 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:50.201459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:50.201428 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:21:50.202169 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:21:50.202140 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75097016_cdfc_4218_8c24_bee5d1bc31a1.slice/crio-e19a67e4a617996ebe26e6ead9a08f7abcab5ce1c724f6252447bcb56ca70707 WatchSource:0}: Error finding container e19a67e4a617996ebe26e6ead9a08f7abcab5ce1c724f6252447bcb56ca70707: Status 404 returned error can't find the container with id e19a67e4a617996ebe26e6ead9a08f7abcab5ce1c724f6252447bcb56ca70707 Oct 31 00:21:51.044409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.044316 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" event={"ID":"75097016-cdfc-4218-8c24-bee5d1bc31a1","Type":"ContainerStarted","Data":"406328ccaba2fe99a9f54f7066948a30340d888e72e0891fd7e8820c4dd2a932"} Oct 31 00:21:51.044409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.044380 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" event={"ID":"75097016-cdfc-4218-8c24-bee5d1bc31a1","Type":"ContainerStarted","Data":"e19a67e4a617996ebe26e6ead9a08f7abcab5ce1c724f6252447bcb56ca70707"} Oct 31 00:21:51.044655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.044530 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:21:51.070554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.070498 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" podStartSLOduration=1.5802301349999999 podStartE2EDuration="2.070484445s" totalImagesPullingTime="490.25431ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:21:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:21:51.069150871 +0000 UTC m=+1633.102246954" watchObservedRunningTime="2026-10-31 00:21:51.070484445 +0000 UTC m=+1633.103580515" Oct 31 00:21:51.727646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.727612 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:21:51.728122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.728101 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:21:51.883318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:51.883291 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:21:57.057926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:21:57.057890 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:22:03.212956 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:03.212916 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:22:09.373816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.373765 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:22:09.382978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.382946 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.417984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.417951 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:22:09.460066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.460033 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:22:09.466112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.466084 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.471253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.471225 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w1-7a7688-openshift-default-dockercfg-c2t4t\"" Oct 31 00:22:09.488514 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.488436 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:22:09.515754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.515709 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dchjj\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.515969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.515801 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.515969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.515841 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.515969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.515909 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.515969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.515953 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.516170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.516148 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.516209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.516179 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.516324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.516303 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.516561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.516537 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.516894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.516866 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:22:09.523398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.523370 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.545851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.544828 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:22:09.618758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618452 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.618758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618538 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.618758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618726 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618814 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618847 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618865 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618914 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618945 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dchjj\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.618970 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619065 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619106 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619299 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619337 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619366 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619405 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619507 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.619736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619546 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.619736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.620006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619830 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619861 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619862 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.619918 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xx7l9\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.620284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620120 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620141 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620185 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.620284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620201 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620276 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.620545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620349 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.620545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620372 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.620545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620439 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.620700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620547 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlpx6\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.620700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.620599 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.621647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.621620 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.622981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.622957 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.654468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.652651 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.654468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.652896 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dchjj\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.716698 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.716667 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722629 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722713 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlpx6\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722744 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722798 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722838 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722888 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722919 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.722948 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723004 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723053 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723103 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723141 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xx7l9\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723419 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723454 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723548 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.723961 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.724063 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.724230 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.724600 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.725809 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.726476 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.727282 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.727943 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.728357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.728202 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.745400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.729457 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.826806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.809526 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.826806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.810305 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.826806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.822705 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.826806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.823510 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.826806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.824383 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlpx6\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.854867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.851628 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xx7l9\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:09.861444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.861423 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:09.863376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:09.863343 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\") pod \"e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:10.115806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.110938 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:10.154305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.154259 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:10.194931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.194875 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:22:10.213113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.213077 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.213222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.213170 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.213291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.213218 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.445657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.445622 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" event={"ID":"0f189999-8354-4304-8558-d88276d509cf","Type":"ContainerStarted","Data":"21b1b766ab017aa0de9846f91f44eab0ae9333994c7c2b38244c127cf8c00ec2"} Oct 31 00:22:10.446316 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.445685 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" event={"ID":"0f189999-8354-4304-8558-d88276d509cf","Type":"ContainerStarted","Data":"638e06d4137157a04a24bb6d05722cf6d23e0c2e734641ff47a2ccb85d646f2f"} Oct 31 00:22:10.601864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.585462 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:22:10.604132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.604068 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podStartSLOduration=1.604042655 podStartE2EDuration="1.604042655s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:10.499425322 +0000 UTC m=+1652.532521394" watchObservedRunningTime="2026-10-31 00:22:10.604042655 +0000 UTC m=+1652.637138743" Oct 31 00:22:10.732346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.732274 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:10.757281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.749227 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.757281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.749323 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.757281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.749364 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.856822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.856760 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.857061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.857040 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.857191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.857176 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:10.880805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.879020 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:22:10.935931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.935891 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11091-3bb012"] Oct 31 00:22:10.943336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.943310 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:10.970531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.970106 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lr8lt\"" Oct 31 00:22:10.975861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.973706 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-3bb012"] Oct 31 00:22:10.987837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:10.987542 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g7gxc\" (UniqueName: \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\") pod \"test-curl-11091-3bb012\" (UID: \"2a3e8672-8a80-41e5-8eae-3432021a8a9e\") " pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:11.105819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.095340 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g7gxc\" (UniqueName: \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\") pod \"test-curl-11091-3bb012\" (UID: \"2a3e8672-8a80-41e5-8eae-3432021a8a9e\") " pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:11.144392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.144354 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g7gxc\" (UniqueName: \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\") pod \"test-curl-11091-3bb012\" (UID: \"2a3e8672-8a80-41e5-8eae-3432021a8a9e\") " pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:11.291394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.291091 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:11.494215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.494174 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" event={"ID":"bea6a00f-672b-4a94-971b-106c506294fd","Type":"ContainerStarted","Data":"7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b"} Oct 31 00:22:11.494685 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.494243 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" event={"ID":"bea6a00f-672b-4a94-971b-106c506294fd","Type":"ContainerStarted","Data":"6be2d848741b09295c23b85059499c3381f89e3212983213ab37b0deb7812411"} Oct 31 00:22:11.499774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.499506 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" event={"ID":"6fa69024-0355-4478-afa2-ef91dffc4adf","Type":"ContainerStarted","Data":"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7"} Oct 31 00:22:11.499774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.499565 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" event={"ID":"6fa69024-0355-4478-afa2-ef91dffc4adf","Type":"ContainerStarted","Data":"8125abdd85d176c5c2f079e124be2515f30727425ef832a08a28b65c5ca554a8"} Oct 31 00:22:11.558292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.558160 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podStartSLOduration=2.5581362 podStartE2EDuration="2.5581362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:11.544160737 +0000 UTC m=+1653.577256808" watchObservedRunningTime="2026-10-31 00:22:11.5581362 +0000 UTC m=+1653.591232330" Oct 31 00:22:11.602864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.591945 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podStartSLOduration=2.591922841 podStartE2EDuration="2.591922841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:11.590303364 +0000 UTC m=+1653.623399434" watchObservedRunningTime="2026-10-31 00:22:11.591922841 +0000 UTC m=+1653.625018912" Oct 31 00:22:11.688099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.688000 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-3bb012"] Oct 31 00:22:11.718610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.718575 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:11.718743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:11.718636 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:12.116060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.115967 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:12.118657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.118328 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:12.118657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.118386 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:12.158809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.158643 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:12.159011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.158819 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:12.159011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.158866 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:12.649820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.640887 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-3bb012" event={"ID":"2a3e8672-8a80-41e5-8eae-3432021a8a9e","Type":"ContainerStarted","Data":"f0df3fe8da7cfd1d18605dee710144e48182fff38edd889ef8cfe7f758fbffba"} Oct 31 00:22:12.744182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.718525 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:12.744182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:12.718586 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:13.120540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.119896 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.120540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.119962 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:13.161812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.157905 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.161812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.157961 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:13.725975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.725941 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:13.726510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.726485 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:13.727428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.727355 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-3bb012" event={"ID":"2a3e8672-8a80-41e5-8eae-3432021a8a9e","Type":"ContainerStarted","Data":"fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d"} Oct 31 00:22:13.771240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:13.771168 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11091-3bb012" podStartSLOduration=2.7305744 podStartE2EDuration="3.771144694s" totalImagesPullingTime="1.040570294s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:13.748032107 +0000 UTC m=+1655.781128178" watchObservedRunningTime="2026-10-31 00:22:13.771144694 +0000 UTC m=+1655.804240765" Oct 31 00:22:14.125856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.124006 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.125856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.124068 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:14.165809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.163347 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.165809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.163431 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:14.729671 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.729392 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:14.732821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:14.731819 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:15.114865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.113801 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.114865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.113866 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:15.169809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.169266 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.169809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.169334 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:15.721315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.721274 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:15.721597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.721558 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:15.976878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.975900 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:22:15.976878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:15.975977 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:22:16.126064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.124290 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.126064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.124351 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:16.155361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.155320 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.155609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.155583 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:16.719372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.719332 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body= Oct 31 00:22:16.719730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:16.719696 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" Oct 31 00:22:17.121991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.118104 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:17.121991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.118165 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:17.157233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.157192 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" start-of-body= Oct 31 00:22:17.190994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.157421 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.122:15021/healthz/ready\": dial tcp 10.134.0.122:15021: connect: connection refused" Oct 31 00:22:17.607493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.606040 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11091-3bb012"] Oct 31 00:22:17.607493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.606249 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11091-3bb012" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" containerName="test-curl-11091-3bb012" containerID="cri-o://fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d" gracePeriod=2 Oct 31 00:22:17.614976 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:17.614053 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-3bb012\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:17.619507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.619475 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11091-3bb012"] Oct 31 00:22:17.737718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.737187 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11091-88ef39"] Oct 31 00:22:17.778862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.738845 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" containerName="test-curl-11091-3bb012" Oct 31 00:22:17.778862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.738873 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" containerName="test-curl-11091-3bb012" Oct 31 00:22:17.778862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.739149 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" containerName="test-curl-11091-3bb012" Oct 31 00:22:17.810202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.810170 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:17.810503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.810468 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-88ef39"] Oct 31 00:22:17.810651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.810485 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:17.811253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.811192 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:17.812833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.812768 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:22:17.814032 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:17.813999 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-3bb012\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:17.895807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.895340 6234 generic.go:356] "Generic (PLEG): container finished" podID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerID="3b11d0dfe4cf3fba1984d0b8cad201ca35d0cdcf9870e33fc14a8d40c2cd7a26" exitCode=0 Oct 31 00:22:17.903811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.896532 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerDied","Data":"3b11d0dfe4cf3fba1984d0b8cad201ca35d0cdcf9870e33fc14a8d40c2cd7a26"} Oct 31 00:22:17.903811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.897014 6234 scope.go:122] "RemoveContainer" containerID="3b11d0dfe4cf3fba1984d0b8cad201ca35d0cdcf9870e33fc14a8d40c2cd7a26" Oct 31 00:22:17.906428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.906362 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:17.906677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.906653 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:17.906844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.906824 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:17.934257 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:17.934212 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-3bb012\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:17.947732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:17.938941 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkfm2\" (UniqueName: \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\") pod \"test-curl-11091-88ef39\" (UID: \"73b2d30d-7011-430f-bc67-63117fc2d89c\") " pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:18.007650 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:18.007603 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-3bb012\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:18.049043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.049006 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkfm2\" (UniqueName: \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\") pod \"test-curl-11091-88ef39\" (UID: \"73b2d30d-7011-430f-bc67-63117fc2d89c\") " pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:18.064515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.064485 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkfm2\" (UniqueName: \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\") pod \"test-curl-11091-88ef39\" (UID: \"73b2d30d-7011-430f-bc67-63117fc2d89c\") " pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:18.089811 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:18.087739 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-3bb012\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:18.120750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.120713 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body= Oct 31 00:22:18.120953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.120772 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" Oct 31 00:22:18.125894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.125552 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:18.268828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.266103 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:18.292810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.282201 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:18.325882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.310062 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:22:18.826721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.826689 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11091-88ef39"] Oct 31 00:22:18.827065 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:22:18.827044 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod73b2d30d_7011_430f_bc67_63117fc2d89c.slice/crio-09f0ca1e9c558cec56c0fbf29267beec77a25eb62b3d3530819f0c09a94c85c8 WatchSource:0}: Error finding container 09f0ca1e9c558cec56c0fbf29267beec77a25eb62b3d3530819f0c09a94c85c8: Status 404 returned error can't find the container with id 09f0ca1e9c558cec56c0fbf29267beec77a25eb62b3d3530819f0c09a94c85c8 Oct 31 00:22:18.904928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.904898 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-88ef39" event={"ID":"73b2d30d-7011-430f-bc67-63117fc2d89c","Type":"ContainerStarted","Data":"09f0ca1e9c558cec56c0fbf29267beec77a25eb62b3d3530819f0c09a94c85c8"} Oct 31 00:22:18.907244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:18.907222 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerStarted","Data":"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383"} Oct 31 00:22:19.118133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.118061 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:19.118517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.118495 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:19.119082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.119062 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:22:19.841154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.841127 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:19.868364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.868323 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:22:19.868636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.868617 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:19.868684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.868659 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:19.916265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.916169 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11091-88ef39" event={"ID":"73b2d30d-7011-430f-bc67-63117fc2d89c","Type":"ContainerStarted","Data":"7f0fe69b5368cf14fd703438017bba4c6eebe722b87069557a7162d880233fcc"} Oct 31 00:22:19.917518 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.917493 6234 generic.go:356] "Generic (PLEG): container finished" podID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" containerID="fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d" exitCode=137 Oct 31 00:22:19.917650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.917526 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-3bb012" Oct 31 00:22:19.917650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.917567 6234 scope.go:122] "RemoveContainer" containerID="fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d" Oct 31 00:22:19.931089 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.931064 6234 scope.go:122] "RemoveContainer" containerID="fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d" Oct 31 00:22:19.931752 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:19.931720 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d\": container with ID starting with fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d not found: ID does not exist" containerID="fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d" Oct 31 00:22:19.931859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.931765 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d"} err="failed to get container status \"fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d\": rpc error: code = NotFound desc = could not find container \"fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d\": container with ID starting with fce622700d0694c163e172d8ba49a3c121915444417d4ce02591a7b1a434c63d not found: ID does not exist" Oct 31 00:22:19.939457 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.939418 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11091-88ef39" podStartSLOduration=2.939406481 podStartE2EDuration="2.939406481s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:19.934722958 +0000 UTC m=+1661.967819029" watchObservedRunningTime="2026-10-31 00:22:19.939406481 +0000 UTC m=+1661.972502550" Oct 31 00:22:19.941248 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.941220 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\" (UniqueName: \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\") pod \"2a3e8672-8a80-41e5-8eae-3432021a8a9e\" (UID: \"2a3e8672-8a80-41e5-8eae-3432021a8a9e\") " Oct 31 00:22:19.943588 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:19.943568 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc" pod "2a3e8672-8a80-41e5-8eae-3432021a8a9e" (UID: "2a3e8672-8a80-41e5-8eae-3432021a8a9e"). InnerVolumeSpecName "kube-api-access-g7gxc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:20.044706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:20.044671 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g7gxc\" (UniqueName: \"kubernetes.io/projected/2a3e8672-8a80-41e5-8eae-3432021a8a9e-kube-api-access-g7gxc\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:20.085367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:20.085336 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a3e8672-8a80-41e5-8eae-3432021a8a9e" path="/var/lib/kubelet/pods/2a3e8672-8a80-41e5-8eae-3432021a8a9e/volumes" Oct 31 00:22:20.868455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:20.868421 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:20.868924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:20.868477 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:21.868769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:21.868727 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:21.869312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:21.868823 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:22.868319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:22.868285 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:22.868510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:22.868337 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:23.868580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:23.868543 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:23.869005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:23.868603 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:24.396945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.396913 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:22:24.403258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.403237 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.420923 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.420898 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458141 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458200 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458282 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8wwth\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458424 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458555 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458601 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458756 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458776 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.459023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.458867 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.464828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.464803 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.473359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.473331 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:22:24.560096 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560061 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560096 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560101 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560159 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560187 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560332 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560436 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8wwth\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560529 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560542 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560625 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560597 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560731 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.560813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560805 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560847 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560874 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.560937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560874 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.561097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560964 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.561097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.560994 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.561097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561011 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.561097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561048 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.561097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561090 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.561362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561129 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x9m9p\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.561577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561505 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.562002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.561978 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.563947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.563921 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.564145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.564129 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.574767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.574733 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.575538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.575508 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8wwth\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.662979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.662874 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.662979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.662937 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.662994 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663033 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663057 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663109 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663212 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x9m9p\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663415 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663487 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.663945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.663925 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.664141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.664122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.664216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.664126 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.664216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.664185 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.665673 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.665655 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.665833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.665774 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.676670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.676646 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.676842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.676774 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x9m9p\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\") pod \"e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.718834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.718806 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:24.783157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.783129 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:24.867891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.867764 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:24.867891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:24.867882 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:25.009644 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:22:25.009598 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b588a2_f55d_4fcc_bf3f_db6834789de6.slice/crio-27e9e4711ac97008636cb653fc2405f2f5926a22bdb55b64a8ec21c7d83c2227 WatchSource:0}: Error finding container 27e9e4711ac97008636cb653fc2405f2f5926a22bdb55b64a8ec21c7d83c2227: Status 404 returned error can't find the container with id 27e9e4711ac97008636cb653fc2405f2f5926a22bdb55b64a8ec21c7d83c2227 Oct 31 00:22:25.010738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.010712 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:22:25.012552 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.012517 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.012645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.012581 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.012645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.012610 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.103049 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:22:25.103013 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4229234f_54c4_40bd_804f_6ce0436a15a0.slice/crio-37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13 WatchSource:0}: Error finding container 37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13: Status 404 returned error can't find the container with id 37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13 Oct 31 00:22:25.104586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.104562 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:22:25.106124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.106092 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.106226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.106152 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.106226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.106186 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:25.868440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.868405 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:25.868598 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.868459 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:25.970149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.970115 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerStarted","Data":"409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13"} Oct 31 00:22:25.970324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.970198 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerStarted","Data":"37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13"} Oct 31 00:22:25.971894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.971870 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerStarted","Data":"a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a"} Oct 31 00:22:25.972021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.971909 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerStarted","Data":"27e9e4711ac97008636cb653fc2405f2f5926a22bdb55b64a8ec21c7d83c2227"} Oct 31 00:22:25.999126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:25.999083 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podStartSLOduration=1.99906922 podStartE2EDuration="1.99906922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:25.994630432 +0000 UTC m=+1668.027726503" watchObservedRunningTime="2026-10-31 00:22:25.99906922 +0000 UTC m=+1668.032165289" Oct 31 00:22:26.019518 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.019465 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podStartSLOduration=2.019451225 podStartE2EDuration="2.019451225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:26.018971807 +0000 UTC m=+1668.052067877" watchObservedRunningTime="2026-10-31 00:22:26.019451225 +0000 UTC m=+1668.052547295" Oct 31 00:22:26.719526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.719492 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:26.721051 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.721022 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:26.721186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.721076 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:26.783748 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.783717 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:26.785369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.785339 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:26.785481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.785392 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:26.868339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.868309 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:26.868515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:26.868370 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:27.719920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.719884 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:27.720294 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.719947 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:27.784475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.784445 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:27.784623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.784494 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:27.867910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.867876 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:27.868072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:27.867941 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:28.719395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.719358 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:28.719575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.719416 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:28.784059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.784024 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:28.784462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.784075 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:28.868661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.868624 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:28.868868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:28.868692 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:29.720034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.719999 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:29.720216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.720058 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:29.783735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.783698 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:29.783949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.783753 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:29.868623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.868587 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:29.869037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.868637 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:29.926775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:29.926746 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:22:30.719713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.719681 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:30.719884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.719739 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:30.784036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.784003 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:30.784210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.784058 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:30.867897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.867863 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:30.868108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.867926 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:30.951824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.951790 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11091-88ef39"] Oct 31 00:22:30.952255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.951973 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11091-88ef39" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" containerName="test-curl-11091-88ef39" containerID="cri-o://7f0fe69b5368cf14fd703438017bba4c6eebe722b87069557a7162d880233fcc" gracePeriod=2 Oct 31 00:22:30.956000 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:30.955953 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-88ef39\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:30.960132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:30.960104 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11091-88ef39"] Oct 31 00:22:31.719570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.719532 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.719770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.719587 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:31.784093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.784058 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.784234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.784113 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:31.868657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.868619 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:31.868863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:31.868678 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:32.720707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.720672 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.721470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.721434 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:32.783974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.783939 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.784142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.784009 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:32.868110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.868076 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:32.868292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:32.868130 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:33.032513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.032480 6234 generic.go:356] "Generic (PLEG): container finished" podID="73b2d30d-7011-430f-bc67-63117fc2d89c" containerID="7f0fe69b5368cf14fd703438017bba4c6eebe722b87069557a7162d880233fcc" exitCode=137 Oct 31 00:22:33.167687 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:33.167650 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:22:33.183774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.183750 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:33.340884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.340761 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\" (UniqueName: \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\") pod \"73b2d30d-7011-430f-bc67-63117fc2d89c\" (UID: \"73b2d30d-7011-430f-bc67-63117fc2d89c\") " Oct 31 00:22:33.343103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.343076 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2" pod "73b2d30d-7011-430f-bc67-63117fc2d89c" (UID: "73b2d30d-7011-430f-bc67-63117fc2d89c"). InnerVolumeSpecName "kube-api-access-bkfm2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:22:33.443126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.443087 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkfm2\" (UniqueName: \"kubernetes.io/projected/73b2d30d-7011-430f-bc67-63117fc2d89c-kube-api-access-bkfm2\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:22:33.720138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.720102 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.720323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.720155 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:33.783836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.783764 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.784289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.783874 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:33.867771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.867735 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:33.867961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:33.867803 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:34.043095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.042996 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:34.043095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.043055 6234 scope.go:122] "RemoveContainer" containerID="7f0fe69b5368cf14fd703438017bba4c6eebe722b87069557a7162d880233fcc" Oct 31 00:22:34.056230 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:22:34.056204 6234 status_manager.go:1045] "Failed to get status for pod" err="pods \"test-curl-11091-88ef39\" is forbidden: User \"system:node:ip-10-0-130-152.us-east-2.compute.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-130-152.us-east-2.compute.internal' and this object" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" pod="opendatahub/test-curl-11091-88ef39" Oct 31 00:22:34.087242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.087209 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" path="/var/lib/kubelet/pods/73b2d30d-7011-430f-bc67-63117fc2d89c/volumes" Oct 31 00:22:34.719452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.719417 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.719635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.719468 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:34.783651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.783617 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.783857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.783670 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:34.867856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.867823 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:34.868030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:34.867875 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:35.719191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.719154 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.719358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.719219 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:35.783530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.783500 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.783709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.783551 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:35.868104 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.868071 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:35.868498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:35.868125 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:36.719394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.719361 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.719554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.719413 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:36.783557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.783524 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.783727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.783577 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:36.868140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.868109 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:36.868522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:36.868164 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:37.719264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.719233 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.719440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.719285 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:37.784385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.784351 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.784556 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.784401 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:37.868471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.868436 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:37.868875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:37.868488 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:38.719649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.719614 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:38.719866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.719675 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:38.783843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.783774 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:38.784025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.783859 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:38.868329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.868295 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:38.868503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:38.868355 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:39.719006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.718972 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:39.719202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.719138 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:39.719202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.719187 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:39.783430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.783401 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:39.785116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.785091 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:39.785226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.785138 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:39.791516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.791490 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:22:39.792766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.792744 6234 cpu_manager.go:414] "containerMap: removing container" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" containerName="test-curl-11091-88ef39" Oct 31 00:22:39.792920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.792769 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" containerName="test-curl-11091-88ef39" Oct 31 00:22:39.793078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.793062 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="73b2d30d-7011-430f-bc67-63117fc2d89c" containerName="test-curl-11091-88ef39" Oct 31 00:22:39.806000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.805976 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.813545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.813514 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-34b786fc-openshift-default-dockercfg-7rgcz\"" Oct 31 00:22:39.815063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.815030 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:22:39.856209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.856156 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.856412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.856364 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.856565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.856546 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lfwhw\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.856770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.856744 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.856891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.856869 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.857078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.857046 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.857265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.857245 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.857355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.857339 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.857429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.857414 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.868276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.868249 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:39.868421 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.868319 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.959704 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.959819 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.959884 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960002 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960092 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960157 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lfwhw\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960269 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960346 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.960408 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.961744 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.963899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.962074 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.992855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.971141 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.992855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.971390 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:39.992855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:39.971630 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.081810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.066489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lfwhw\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.081810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.070278 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.130367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.107386 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.130367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.107985 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.144858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.144247 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:40.663968 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:22:40.663935 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb4ab648a_8532_4efa_a68d_1748e5d1bcd8.slice/crio-a9315ea50021ce5600b7a1660a24e0dc825ab29fe86dd321f4ff46f2211659a5 WatchSource:0}: Error finding container a9315ea50021ce5600b7a1660a24e0dc825ab29fe86dd321f4ff46f2211659a5: Status 404 returned error can't find the container with id a9315ea50021ce5600b7a1660a24e0dc825ab29fe86dd321f4ff46f2211659a5 Oct 31 00:22:40.665015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.664989 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:22:40.667448 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.667413 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:40.667537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.667481 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:40.667537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.667507 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:40.719126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.719099 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:40.719268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.719144 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:40.783704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.783663 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:40.783886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.783723 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:40.868605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.868570 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:40.868824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:40.868622 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:41.172148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.172112 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerStarted","Data":"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8"} Oct 31 00:22:41.172615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.172171 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerStarted","Data":"a9315ea50021ce5600b7a1660a24e0dc825ab29fe86dd321f4ff46f2211659a5"} Oct 31 00:22:41.201852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.201778 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podStartSLOduration=2.201763596 podStartE2EDuration="2.201763596s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:22:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:22:41.199657333 +0000 UTC m=+1683.232753404" watchObservedRunningTime="2026-10-31 00:22:41.201763596 +0000 UTC m=+1683.234859666" Oct 31 00:22:41.719259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.719220 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:41.719468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.719278 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:41.783956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.783923 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:41.784101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.783975 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:41.867849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.867814 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:41.868052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:41.867873 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:42.145535 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.145460 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:42.151115 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.151088 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:42.181073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.181050 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:42.181994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.181975 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:22:42.719772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.719741 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:42.719977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.719824 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:42.783970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.783941 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:42.784144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.783993 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:42.867947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.867909 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:42.868119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:42.867968 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:43.719034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.719002 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:43.719406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.719052 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:43.783485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.783446 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:43.783653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.783503 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:43.868109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.868073 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:43.868285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:43.868127 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:44.720069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.720033 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:44.720473 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.720086 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:44.784221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.784183 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:44.784391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.784243 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:44.868488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.868451 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:44.868638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:44.868502 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:45.719706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.719667 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:45.719904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.719723 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:45.783482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.783448 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:45.783898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.783494 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:45.868771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.868737 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:45.868771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:45.868802 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:46.719474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.719441 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:46.719631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.719493 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:46.783974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.783940 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:46.784350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.783989 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:46.867926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.867894 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:46.868070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:46.867947 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:47.720010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.719974 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:47.720194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.720026 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:47.784388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.784353 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:47.784841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.784404 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:47.868733 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.868697 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:47.868962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:47.868753 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:48.719579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.719541 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:48.719762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.719595 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:48.784396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.784357 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:48.784918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.784411 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:48.868296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.868253 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:48.868492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.868312 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:48.868492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.868360 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:22:48.869328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.869019 6234 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383"} pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 31 00:22:48.869328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:48.869066 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" containerID="cri-o://24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383" gracePeriod=30 Oct 31 00:22:49.719614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:49.719579 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:49.719808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:49.719637 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:49.784075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:49.784036 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:49.784248 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:49.784100 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:50.719080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:50.719048 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:50.719550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:50.719136 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:50.783492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:50.783460 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:50.783634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:50.783520 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:51.719822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:51.719746 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:51.720262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:51.719834 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:51.783994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:51.783956 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:51.784168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:51.784015 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:52.719377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:52.719341 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:52.719541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:52.719397 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:52.784397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:52.784356 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:52.784794 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:52.784418 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:53.719558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.719522 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:53.719737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.719586 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:53.783993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.783953 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:53.784128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.784021 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:53.984390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.984347 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:53.984702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.984454 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:53.984702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:53.984506 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:22:54.566083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.566047 6234 generic.go:356] "Generic (PLEG): container finished" podID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerID="24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383" exitCode=0 Oct 31 00:22:54.566298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.566277 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerDied","Data":"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383"} Oct 31 00:22:54.566496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.566478 6234 scope.go:122] "RemoveContainer" containerID="3b11d0dfe4cf3fba1984d0b8cad201ca35d0cdcf9870e33fc14a8d40c2cd7a26" Oct 31 00:22:54.720097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.720060 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:54.720261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.720124 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:54.783656 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.783623 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:54.783848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:54.783677 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:55.576378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.576341 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerStarted","Data":"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da"} Oct 31 00:22:55.719577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.719542 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:22:55.719730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.719596 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:22:55.719730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.719641 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:22:55.720292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.720272 6234 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a"} pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 31 00:22:55.720341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.720321 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" containerID="cri-o://a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a" gracePeriod=30 Oct 31 00:22:55.784217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.784183 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:22:55.784380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.784231 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:22:55.784380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.784269 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:22:55.784737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.784719 6234 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="istio-proxy" containerStatusID={"Type":"cri-o","ID":"409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13"} pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" containerMessage="Container istio-proxy failed startup probe, will be restarted" Oct 31 00:22:55.784774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.784761 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" containerID="cri-o://409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13" gracePeriod=30 Oct 31 00:22:55.867925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.867854 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:22:55.868067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.867985 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:55.868067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:55.868048 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:56.868362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:56.868333 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:56.868733 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:56.868390 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:57.868668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:57.868635 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:57.869096 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:57.868688 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:58.868662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:58.868627 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:58.868884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:58.868690 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:59.868697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:59.868659 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:22:59.869049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:59.868721 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:22:59.927122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:22:59.927091 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:23:00.868674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.868641 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:00.868903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.868694 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:00.893186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.893147 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:00.893476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.893230 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:00.893476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.893264 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:00.903977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.903945 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:00.904073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.904022 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:00.904073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:00.904054 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:23:01.627025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.626995 6234 generic.go:356] "Generic (PLEG): container finished" podID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerID="a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a" exitCode=0 Oct 31 00:23:01.627236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.627073 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerDied","Data":"a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a"} Oct 31 00:23:01.627236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.627134 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerStarted","Data":"fe9b29b888031720e605ff106b29e4122ed1ef267a002b46770377412b4c9c92"} Oct 31 00:23:01.629300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.629277 6234 generic.go:356] "Generic (PLEG): container finished" podID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerID="409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13" exitCode=0 Oct 31 00:23:01.629417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.629348 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerDied","Data":"409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13"} Oct 31 00:23:01.629417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.629390 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerStarted","Data":"81a7c8b8fbf3008a7d03ea62f077c220df3dba4da8666f8646cd90dfd5c0e03a"} Oct 31 00:23:01.719057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.719020 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:23:01.719262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.719182 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.719262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.719232 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:01.784193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.784153 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:23:01.784423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.784399 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.784503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.784458 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:01.868487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.868454 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:01.868661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:01.868511 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:02.719646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.719605 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.720088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.719665 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:02.783574 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.783536 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.783715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.783589 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:02.868304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.868267 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:02.868526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:02.868330 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:03.163718 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:23:03.163631 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:23:03.719494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.719454 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.719685 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.719525 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:03.783712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.783675 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.783951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.783750 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:03.868520 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.868482 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:03.868697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:03.868543 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:04.719655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.719616 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.719888 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.719686 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:04.784047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.784012 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.784242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.784068 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:04.867937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.867903 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:04.868115 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:04.867968 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:05.719046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.719012 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.719222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.719059 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:05.787214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.787178 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.787868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.787836 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:05.874207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.874171 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:05.874473 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:05.874445 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:06.833984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.833951 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.834758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.834725 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:06.864481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.855312 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.864481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.855385 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:06.925128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.924988 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:06.925128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:06.925062 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:07.727796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:07.727745 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:07.751440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:07.751400 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:07.875811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:07.853361 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:07.899389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:07.899139 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:07.925821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:07.898918 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.014071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.013968 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:08.723277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.723218 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.723504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.723282 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:08.814810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.784652 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.831954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.831906 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:08.872389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.872341 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:08.879587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:08.879202 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:09.721716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.721446 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.721716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.721502 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:09.723006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.722453 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:23:09.879420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.869057 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:23:09.879420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.871324 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.879420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.871371 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:09.900925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.887539 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:09.900925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:09.887593 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:10.362811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.349714 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.437416 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.437371 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.466998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.466237 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.524649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.524603 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:23:10.525264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.525237 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:23:10.534840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.530945 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:23:10.727287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.727248 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.727573 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.727548 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.125:15021/healthz/ready\": dial tcp 10.134.0.125:15021: connect: connection refused" Oct 31 00:23:10.752817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.737161 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:23:10.785816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.785165 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.785816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.785229 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" Oct 31 00:23:10.883809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.871634 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:10.883809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:10.871720 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:11.724482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.724455 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:23:11.728326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.728302 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:23:11.790340 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.790270 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:23:11.791946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.791925 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:23:11.892006 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.891974 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body= Oct 31 00:23:11.892809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:11.892564 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" Oct 31 00:23:12.873453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:12.873419 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:23:12.875049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:12.875020 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:23:33.162831 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:23:33.162770 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:23:54.024453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.024422 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:23:54.030410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.030383 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.037720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.037677 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-34b786fc\"" Oct 31 00:23:54.041219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.041196 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-34b786fc-dockercfg-ldl6l\"" Oct 31 00:23:54.053967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.053934 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:23:54.069317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.068957 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.075338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.069768 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:23:54.096701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.096671 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:23:54.244812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.241679 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.244812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.241753 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dv8g\" (UniqueName: \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.244812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.241876 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.244812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.242196 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qqnjl\" (UniqueName: \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.375681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.375576 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qqnjl\" (UniqueName: \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.395810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.376209 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.396268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.396230 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dv8g\" (UniqueName: \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.396891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.396861 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.410681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.410372 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.476742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.453010 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.555858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.546987 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dv8g\" (UniqueName: \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\") pod \"payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.602851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.595706 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qqnjl\" (UniqueName: \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\") pod \"payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.662567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.662472 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:54.839167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.839132 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:54.971503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:54.971386 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:23:55.168553 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:23:55.168513 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653842f3_cb9b_4827_901b_8562e264027f.slice/crio-1bc3a3cce21862c89a54154550c2ed7fefaf188edd104851e004ed5e8c2f73da WatchSource:0}: Error finding container 1bc3a3cce21862c89a54154550c2ed7fefaf188edd104851e004ed5e8c2f73da: Status 404 returned error can't find the container with id 1bc3a3cce21862c89a54154550c2ed7fefaf188edd104851e004ed5e8c2f73da Oct 31 00:23:55.169723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.169697 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:23:55.643648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.643555 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" event={"ID":"653842f3-cb9b-4827-901b-8562e264027f","Type":"ContainerStarted","Data":"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f"} Oct 31 00:23:55.643648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.643619 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" event={"ID":"653842f3-cb9b-4827-901b-8562e264027f","Type":"ContainerStarted","Data":"1bc3a3cce21862c89a54154550c2ed7fefaf188edd104851e004ed5e8c2f73da"} Oct 31 00:23:55.643859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.643650 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:23:55.645359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.645334 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" event={"ID":"12d742a3-3e30-4cb7-8839-68805e36ed3f","Type":"ContainerStarted","Data":"8521f6179d57f20bf27c6bcb59c02c8a0e28699fb33f860685fdd78ce24a2100"} Oct 31 00:23:55.645476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.645375 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" event={"ID":"12d742a3-3e30-4cb7-8839-68805e36ed3f","Type":"ContainerStarted","Data":"4f4626bd755ddfa4c1c3bb331e01e79bab7243ca26032856120dbf0fb3eaaa51"} Oct 31 00:23:55.645476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.645433 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:23:55.670258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.670209 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" podStartSLOduration=1.670195848 podStartE2EDuration="1.670195848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:55.666611636 +0000 UTC m=+1757.699707707" watchObservedRunningTime="2026-10-31 00:23:55.670195848 +0000 UTC m=+1757.703291919" Oct 31 00:23:55.690151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:23:55.690105 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" podStartSLOduration=2.6900937689999997 podStartE2EDuration="2.690093769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:23:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:23:55.686064009 +0000 UTC m=+1757.719160081" watchObservedRunningTime="2026-10-31 00:23:55.690093769 +0000 UTC m=+1757.723189838" Oct 31 00:24:03.163153 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:03.163113 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:24:06.657677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:06.657642 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:24:06.658214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:06.657926 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:24:32.407809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.368467 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:32.714817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.521656 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:24:32.714817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.524401 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:32.714817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.544881 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:32.714817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.544942 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:24:32.714817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:32.552879 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:33.204811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.200838 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:33.424824 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:33.424749 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:24:33.745813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.742935 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:24:33.774811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.759890 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:24:33.774811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.762097 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:24:33.774811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.765262 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:33.871809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.870249 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:33.871809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.870655 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:33.893823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.893151 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:24:33.893823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.893236 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:24:33.935812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.934975 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:24:33.935812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.935393 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:24:33.936110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.936002 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:24:33.936110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:33.936062 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:24:34.108867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.107553 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:34.135810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.128849 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:34.182813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.164077 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:24:34.761701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.761653 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:34.764002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.763969 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:34.781804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.781733 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:24:34.978838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:34.955693 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:24:35.048418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.009427 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:35.050257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.009471 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:24:35.133846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.127231 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:35.154833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.148493 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": context deadline exceeded" Oct 31 00:24:35.250898 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:35.250848 6234 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.087s" Oct 31 00:24:35.251456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.251427 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:24:35.251661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.251633 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:24:35.263657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.263636 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:24:35.264270 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.264235 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:24:35.414865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.389456 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:35.537458 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.454184 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:35.829836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.792060 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:24:35.833660 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.810155 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:35.849142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:35.833247 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:36.007457 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.007426 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:24:36.026123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.026081 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:24:36.036558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.036527 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:24:36.060813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.059278 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:24:36.127963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.127858 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:36.169196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.168216 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:24:36.183825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.181745 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:24:36.183825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.182095 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:24:36.185800 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.185757 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:24:36.190828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.190809 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:24:36.195022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.194999 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:24:36.214022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:36.213995 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:24:43.126627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.126535 6234 generic.go:356] "Generic (PLEG): container finished" podID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerID="ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363" exitCode=0 Oct 31 00:24:43.127082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.126610 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerDied","Data":"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363"} Oct 31 00:24:43.127178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.127159 6234 scope.go:122] "RemoveContainer" containerID="ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363" Oct 31 00:24:43.129693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.129658 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:24:43.129808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.129755 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:24:43.129859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:43.129826 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:24:44.137971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:44.137938 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerStarted","Data":"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a"} Oct 31 00:24:45.049869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:45.049828 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:24:45.050083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:45.050032 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:45.050168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:45.050082 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:46.049348 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:46.049307 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:46.049891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:46.049361 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:47.049379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:47.049343 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:47.049866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:47.049395 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:48.049385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:48.049299 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:48.049385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:48.049356 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:49.006428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.004890 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:24:49.006428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.005640 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" containerID="cri-o://6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228" gracePeriod=30 Oct 31 00:24:49.009276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.009139 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:24:49.010629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.009436 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" containerID="cri-o://ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286" gracePeriod=30 Oct 31 00:24:49.050927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.049988 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:49.050927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.050043 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:49.124178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.124141 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:24:49.124413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.124384 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" containerID="cri-o://f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8" gracePeriod=30 Oct 31 00:24:49.129853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.129823 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:24:49.130060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:49.130035 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" containerID="cri-o://87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3" gracePeriod=30 Oct 31 00:24:50.049402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:50.049362 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:50.049607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:50.049420 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:51.049319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:51.049285 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:51.049857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:51.049351 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:51.286849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:51.286777 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:24:52.049295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:52.049260 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:52.049479 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:52.049327 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:53.069279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:53.067917 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:53.070090 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:53.070046 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:54.053010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.049914 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:54.053010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.049975 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:54.281964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.281867 6234 generic.go:356] "Generic (PLEG): container finished" podID="1855f8df-d794-4f19-a790-5545eb7e1222" containerID="6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228" exitCode=0 Oct 31 00:24:54.282368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.282167 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" event={"ID":"1855f8df-d794-4f19-a790-5545eb7e1222","Type":"ContainerDied","Data":"6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228"} Oct 31 00:24:54.436323 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:54.436268 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1855f8df_d794_4f19_a790_5545eb7e1222.slice/crio-conmon-6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a67dd4_224d_4701_84a3_48540c600bfe.slice/crio-conmon-ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod254ed5a9_9a0f_4918_b81e_6f71adcb8dea.slice/crio-conmon-f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48a67dd4_224d_4701_84a3_48540c600bfe.slice/crio-ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05affd62_3cea_4900_a6e0_cb30d73b1948.slice/crio-87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1855f8df_d794_4f19_a790_5545eb7e1222.slice/crio-6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05affd62_3cea_4900_a6e0_cb30d73b1948.slice/crio-conmon-87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:24:54.583301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.583267 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:24:54.595544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.595516 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:24:54.598776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.598750 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.598776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.598826 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.598975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.598883 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.598975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.598935 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.598979 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599045 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599137 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599187 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599239 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599284 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599353 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599383 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599380 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599488 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599510 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.599591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599593 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.600156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599671 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.600156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599802 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.600156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599885 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.600156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.599974 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\") pod \"1855f8df-d794-4f19-a790-5545eb7e1222\" (UID: \"1855f8df-d794-4f19-a790-5545eb7e1222\") " Oct 31 00:24:54.600156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.600048 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\") pod \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\" (UID: \"254ed5a9-9a0f-4918-b81e-6f71adcb8dea\") " Oct 31 00:24:54.600625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.600318 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.600625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.600564 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.601356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.600806 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.601356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601091 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.601356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601095 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.601356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601172 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.601356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601242 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.601904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601879 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601915 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601930 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601944 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1855f8df-d794-4f19-a790-5545eb7e1222-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601958 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601973 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.601986 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.602000 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.602014 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.602065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.602027 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.603572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.603548 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "kube-api-access-68kmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.604361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.604050 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:24:54.604361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.604287 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.604361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.604317 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "kube-api-access-zkfxk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.604603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.604397 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token" pod "254ed5a9-9a0f-4918-b81e-6f71adcb8dea" (UID: "254ed5a9-9a0f-4918-b81e-6f71adcb8dea"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.605194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.605168 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.605921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.605896 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.606508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.606490 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo" pod "1855f8df-d794-4f19-a790-5545eb7e1222" (UID: "1855f8df-d794-4f19-a790-5545eb7e1222"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:24:54.634266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.634183 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:24:54.635474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635437 6234 cpu_manager.go:414] "containerMap: removing container" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" Oct 31 00:24:54.635474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635464 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" Oct 31 00:24:54.635745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635524 6234 cpu_manager.go:414] "containerMap: removing container" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" Oct 31 00:24:54.635745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635532 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" Oct 31 00:24:54.635884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635821 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" containerName="istio-proxy" Oct 31 00:24:54.635884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.635858 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerName="istio-proxy" Oct 31 00:24:54.671808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.667073 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.684985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.684954 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-719e12-dockercfg-s7nkh\"" Oct 31 00:24:54.685521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.685490 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-719e12\"" Oct 31 00:24:54.707503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.707468 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwp5r\" (UniqueName: \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.708082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.708043 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.708647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.708627 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.708819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.708802 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zkfxk\" (UniqueName: \"kubernetes.io/projected/1855f8df-d794-4f19-a790-5545eb7e1222-kube-api-access-zkfxk\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.708953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.708939 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.709078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.709063 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1855f8df-d794-4f19-a790-5545eb7e1222-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.709198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.709184 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1855f8df-d794-4f19-a790-5545eb7e1222-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.709308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.709294 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.709418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.709405 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.709539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.709526 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-68kmt\" (UniqueName: \"kubernetes.io/projected/254ed5a9-9a0f-4918-b81e-6f71adcb8dea-kube-api-access-68kmt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.716639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.716614 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:24:54.778683 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.778654 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:24:54.781153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.781130 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:24:54.810340 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810300 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.810530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810362 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810392 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810427 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810481 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.810530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810504 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.810820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810555 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810619 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810798 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810839 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.810991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810891 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.810991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.810938 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.811147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811046 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.811147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811106 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.811252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811189 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.811252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811228 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\") pod \"05affd62-3cea-4900-a6e0-cb30d73b1948\" (UID: \"05affd62-3cea-4900-a6e0-cb30d73b1948\") " Oct 31 00:24:54.811364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811314 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.811420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811374 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\") pod \"48a67dd4-224d-4701-84a3-48540c600bfe\" (UID: \"48a67dd4-224d-4701-84a3-48540c600bfe\") " Oct 31 00:24:54.811677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.811652 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.812418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.812107 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwp5r\" (UniqueName: \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.812963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.812937 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.813438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.813415 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.814121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.814094 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.814411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.814391 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:24:54.814881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.814645 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.815169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.815018 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.815489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.815261 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.816862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.816839 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.817234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.817214 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.817487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.817472 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.817735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.817721 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.856376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.856341 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:24:54.858725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.858695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.871385 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.873901 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.873962 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "kube-api-access-zswk9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.877013 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh" pod "48a67dd4-224d-4701-84a3-48540c600bfe" (UID: "48a67dd4-224d-4701-84a3-48540c600bfe"). InnerVolumeSpecName "kube-api-access-f6psh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.877121 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:24:54.877732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.877690 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token" pod "05affd62-3cea-4900-a6e0-cb30d73b1948" (UID: "05affd62-3cea-4900-a6e0-cb30d73b1948"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:24:54.879934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.879879 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwp5r\" (UniqueName: \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\") pod \"payload-processing-e2e-api-a-719e12-64c9c9475c-jst28\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914384 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zswk9\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-kube-api-access-zswk9\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914415 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914424 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914433 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914444 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914452 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914460 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/48a67dd4-224d-4701-84a3-48540c600bfe-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914468 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/05affd62-3cea-4900-a6e0-cb30d73b1948-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914476 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/48a67dd4-224d-4701-84a3-48540c600bfe-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914483 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f6psh\" (UniqueName: \"kubernetes.io/projected/48a67dd4-224d-4701-84a3-48540c600bfe-kube-api-access-f6psh\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914491 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914499 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914508 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914519 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914527 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914535 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/48a67dd4-224d-4701-84a3-48540c600bfe-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914543 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:54.915992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:54.914552 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/05affd62-3cea-4900-a6e0-cb30d73b1948-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:24:55.013809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.013751 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:55.050486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.050417 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:55.051199 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.051137 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:55.415328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.415290 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" event={"ID":"1855f8df-d794-4f19-a790-5545eb7e1222","Type":"ContainerDied","Data":"ae5a86cd209d9021726eb19ca1117114f55da11d7f9b7274b93e14770b67f471"} Oct 31 00:24:55.420204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.420177 6234 scope.go:122] "RemoveContainer" containerID="6c329bd80e842b1cb6a875794ab9500bbdeaa270b973d4ca1eb54eff0a82b228" Oct 31 00:24:55.420506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.415310 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd" Oct 31 00:24:55.429339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.429309 6234 generic.go:356] "Generic (PLEG): container finished" podID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" containerID="f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8" exitCode=0 Oct 31 00:24:55.429605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.429582 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" event={"ID":"254ed5a9-9a0f-4918-b81e-6f71adcb8dea","Type":"ContainerDied","Data":"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8"} Oct 31 00:24:55.429936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.429749 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" event={"ID":"254ed5a9-9a0f-4918-b81e-6f71adcb8dea","Type":"ContainerDied","Data":"61ae280cd3ce1d8fa494d23db6809e7cc3a31a56f576a76a800cdc31783e1316"} Oct 31 00:24:55.430182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.430159 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq" Oct 31 00:24:55.432818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.432774 6234 generic.go:356] "Generic (PLEG): container finished" podID="48a67dd4-224d-4701-84a3-48540c600bfe" containerID="ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286" exitCode=0 Oct 31 00:24:55.432991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.432975 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" Oct 31 00:24:55.433062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.433021 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" event={"ID":"48a67dd4-224d-4701-84a3-48540c600bfe","Type":"ContainerDied","Data":"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286"} Oct 31 00:24:55.433139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.433062 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc" event={"ID":"48a67dd4-224d-4701-84a3-48540c600bfe","Type":"ContainerDied","Data":"b3dbf1b36b576ad03d1512f468ae6e091999be25cfb8fff2ebb237863afbd62e"} Oct 31 00:24:55.434709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.434688 6234 generic.go:356] "Generic (PLEG): container finished" podID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerID="87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3" exitCode=0 Oct 31 00:24:55.435046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.435029 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" event={"ID":"05affd62-3cea-4900-a6e0-cb30d73b1948","Type":"ContainerDied","Data":"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3"} Oct 31 00:24:55.435175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.435162 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" event={"ID":"05affd62-3cea-4900-a6e0-cb30d73b1948","Type":"ContainerDied","Data":"0d214936bedef686d7e9a127413af24f5090e6f3e97cce3645c9fd37619ea01d"} Oct 31 00:24:55.435324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.435312 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt" Oct 31 00:24:55.449107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.448074 6234 scope.go:122] "RemoveContainer" containerID="f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8" Oct 31 00:24:55.491460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.491434 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:24:55.492873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.492857 6234 scope.go:122] "RemoveContainer" containerID="f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8" Oct 31 00:24:55.493215 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:55.493193 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8\": container with ID starting with f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8 not found: ID does not exist" containerID="f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8" Oct 31 00:24:55.493288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.493223 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8"} err="failed to get container status \"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8\": rpc error: code = NotFound desc = could not find container \"f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8\": container with ID starting with f3189d117511b6d78cd82f498f9b3616f7f05be6200e541ade28dbda67581cc8 not found: ID does not exist" Oct 31 00:24:55.493288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.493242 6234 scope.go:122] "RemoveContainer" containerID="ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286" Oct 31 00:24:55.499012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.498989 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-4s7xd"] Oct 31 00:24:55.504451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.504428 6234 scope.go:122] "RemoveContainer" containerID="ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286" Oct 31 00:24:55.504721 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:55.504700 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286\": container with ID starting with ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286 not found: ID does not exist" containerID="ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286" Oct 31 00:24:55.504840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.504733 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286"} err="failed to get container status \"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286\": rpc error: code = NotFound desc = could not find container \"ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286\": container with ID starting with ab4732c53ffb2a5d636a51b996bbfaea26c6843b98c85e7523d8986794c6d286 not found: ID does not exist" Oct 31 00:24:55.504840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.504752 6234 scope.go:122] "RemoveContainer" containerID="87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3" Oct 31 00:24:55.514018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.513955 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:24:55.516664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.516642 6234 scope.go:122] "RemoveContainer" containerID="87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3" Oct 31 00:24:55.517015 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:55.516993 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3\": container with ID starting with 87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3 not found: ID does not exist" containerID="87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3" Oct 31 00:24:55.517094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.517025 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3"} err="failed to get container status \"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3\": rpc error: code = NotFound desc = could not find container \"87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3\": container with ID starting with 87769e40eb1bd090aca33767f18d7d9daafa703dca41c9d55e43bf0e65e8a8f3 not found: ID does not exist" Oct 31 00:24:55.520738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.520713 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-22vxc"] Oct 31 00:24:55.534022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.533996 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:24:55.558800 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.558746 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-9h5nt"] Oct 31 00:24:55.571182 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:24:55.571153 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod21fb8bd9_d844_4fb8_bb6b_8029479d3f4b.slice/crio-147302fa38932c40b8a2b51e7267ad31215d2eef1ce6b0eec7f2468d98601753 WatchSource:0}: Error finding container 147302fa38932c40b8a2b51e7267ad31215d2eef1ce6b0eec7f2468d98601753: Status 404 returned error can't find the container with id 147302fa38932c40b8a2b51e7267ad31215d2eef1ce6b0eec7f2468d98601753 Oct 31 00:24:55.571354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.571337 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:24:55.576768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.576749 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:24:55.580761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:55.580739 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-69kvq"] Oct 31 00:24:56.062808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.058417 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:56.062808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.058529 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:56.092847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.092808 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" path="/var/lib/kubelet/pods/05affd62-3cea-4900-a6e0-cb30d73b1948/volumes" Oct 31 00:24:56.093771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.093481 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1855f8df-d794-4f19-a790-5545eb7e1222" path="/var/lib/kubelet/pods/1855f8df-d794-4f19-a790-5545eb7e1222/volumes" Oct 31 00:24:56.094130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.094113 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="254ed5a9-9a0f-4918-b81e-6f71adcb8dea" path="/var/lib/kubelet/pods/254ed5a9-9a0f-4918-b81e-6f71adcb8dea/volumes" Oct 31 00:24:56.095023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.095004 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" path="/var/lib/kubelet/pods/48a67dd4-224d-4701-84a3-48540c600bfe/volumes" Oct 31 00:24:56.470385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.470343 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" event={"ID":"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b","Type":"ContainerStarted","Data":"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0"} Oct 31 00:24:56.470841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.470399 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" event={"ID":"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b","Type":"ContainerStarted","Data":"147302fa38932c40b8a2b51e7267ad31215d2eef1ce6b0eec7f2468d98601753"} Oct 31 00:24:56.470841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.470435 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:24:56.534425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:56.534364 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" podStartSLOduration=25.534346019 podStartE2EDuration="25.534346019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:24:56.533008326 +0000 UTC m=+1818.566104396" watchObservedRunningTime="2026-10-31 00:24:56.534346019 +0000 UTC m=+1818.567442093" Oct 31 00:24:57.065029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:57.063111 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:57.065029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:57.063172 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:58.049347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:58.049310 6234 patch_prober.go:29] interesting pod/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" start-of-body= Oct 31 00:24:58.049860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:58.049368 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.114:15021/healthz/ready\": dial tcp 10.134.0.114:15021: connect: connection refused" Oct 31 00:24:59.055254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.055221 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:24:59.056413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.056390 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:24:59.191829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.191748 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:24:59.193067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193042 6234 cpu_manager.go:414] "containerMap: removing container" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" Oct 31 00:24:59.193067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193066 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" Oct 31 00:24:59.193254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193127 6234 cpu_manager.go:414] "containerMap: removing container" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" Oct 31 00:24:59.193254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193137 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" Oct 31 00:24:59.193381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193365 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="48a67dd4-224d-4701-84a3-48540c600bfe" containerName="istio-proxy" Oct 31 00:24:59.193432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.193396 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="05affd62-3cea-4900-a6e0-cb30d73b1948" containerName="istio-proxy" Oct 31 00:24:59.198117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.198093 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.205383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.205352 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-945abd-serving-cert\"" Oct 31 00:24:59.213838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.213812 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:24:59.366861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.348456 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.367245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.367208 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j8xvx\" (UniqueName: \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.367525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.367503 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.373814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.367838 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.478379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.478335 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.484982 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.485085 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j8xvx\" (UniqueName: \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.485151 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:59.485366 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-serving-cert: secret "maas-api-e2e-shared-a-w5-945abd-serving-cert" not found Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:24:59.485441 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls podName:a2492e3e-292f-4eeb-93b6-fcf3ac4b4638 nodeName:}" failed. No retries permitted until 2026-10-31 00:24:59.985420033 +0000 UTC m=+1822.018516104 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls") pod "maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638") : secret "maas-api-e2e-shared-a-w5-945abd-serving-cert" not found Oct 31 00:24:59.493813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.485724 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.499550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.499420 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:24:59.505592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:24:59.505559 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j8xvx\" (UniqueName: \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:00.156700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.155605 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:00.156700 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:25:00.155875 6234 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-serving-cert: secret "maas-api-e2e-shared-a-w5-945abd-serving-cert" not found Oct 31 00:25:00.156700 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:25:00.155926 6234 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls podName:a2492e3e-292f-4eeb-93b6-fcf3ac4b4638 nodeName:}" failed. No retries permitted until 2026-10-31 00:25:01.155912904 +0000 UTC m=+1823.189008952 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls") pod "maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638") : secret "maas-api-e2e-shared-a-w5-945abd-serving-cert" not found Oct 31 00:25:00.783328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.783290 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:25:00.788869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.788844 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.798967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.798940 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w4-a39be4-openshift-default-dockercfg-clxb8\"" Oct 31 00:25:00.802029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.801995 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:25:00.807131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.807107 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:25:00.807255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.807222 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.833132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.833098 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:25:00.892308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892266 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892330 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892574 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892550 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892591 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-725qg\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892668 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892725 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892856 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.892954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.892896 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.893017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.893000 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995137 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995275 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995313 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-725qg\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995339 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vr5d4\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995367 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995492 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995451 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995510 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995553 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995615 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995656 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995797 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995836 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.995985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995885 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.995985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.995936 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.996140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996032 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996173 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.996255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996232 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996268 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996318 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996324 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:00.996657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996631 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996661 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.996850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.996748 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.998092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.998069 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:00.998627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:00.998604 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:01.008540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.008516 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:01.008938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.008916 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-725qg\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:01.098176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098085 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098243 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098301 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098326 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vr5d4\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098426 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098504 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098626 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098652 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.098974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.098948 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.099119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.099013 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.099119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.099064 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.099405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.099130 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.099405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.099164 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.101061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.101039 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.101156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.101140 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.109724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.109699 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:01.113382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.113357 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.113511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.113407 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vr5d4\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\") pod \"e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.144290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.144257 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:01.205846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.205435 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:01.221869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.221093 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:01.358890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.358849 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:01.449608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.449562 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:25:01.458997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.458952 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:01.459281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.459253 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:01.459451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.459434 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:01.726809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.687869 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" event={"ID":"0f45d136-993a-4d0f-a23b-0531bf9077bc","Type":"ContainerStarted","Data":"9d20938719159fc8e03de26a75e93416d9e1d77c1d17c2bc10ec704cfc45edf9"} Oct 31 00:25:01.923423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.922806 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:25:01.938100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.938054 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:01.938243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.938137 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:01.938243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:01.938174 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:02.163336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:02.163309 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:25:02.971821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:02.967937 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" event={"ID":"0f45d136-993a-4d0f-a23b-0531bf9077bc","Type":"ContainerStarted","Data":"f14f00fd1752376e60f36ab90b581aa390cdd27e4d604fc4eb80dbf33c645546"} Oct 31 00:25:02.999809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:02.995743 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" event={"ID":"956df451-22ed-4ae2-b363-df35b53bfc73","Type":"ContainerStarted","Data":"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13"} Oct 31 00:25:03.000412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.000378 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" event={"ID":"956df451-22ed-4ae2-b363-df35b53bfc73","Type":"ContainerStarted","Data":"2e6d6cc15f73cbb46f8222aa388b0d3ec0f4e53721834646295e86a177e9648f"} Oct 31 00:25:03.000584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.000564 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" event={"ID":"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638","Type":"ContainerStarted","Data":"0ccd76327ea3d3f33b8c79f50eb580758aa20ffc61d08341c3e12658c9142d78"} Oct 31 00:25:03.094810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.091052 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podStartSLOduration=3.09103103 podStartE2EDuration="3.09103103s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:03.026092547 +0000 UTC m=+1825.059188621" watchObservedRunningTime="2026-10-31 00:25:03.09103103 +0000 UTC m=+1825.124127100" Oct 31 00:25:03.111287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.111254 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:03.111625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.111604 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:03.111775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.111751 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:03.191484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.191311 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:03.192875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.192816 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:03.193036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.192900 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:03.222852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:03.222696 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podStartSLOduration=3.222673661 podStartE2EDuration="3.222673661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:03.163861509 +0000 UTC m=+1825.196957582" watchObservedRunningTime="2026-10-31 00:25:03.222673661 +0000 UTC m=+1825.255769735" Oct 31 00:25:03.313078 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:25:03.308906 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:25:04.134336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.134302 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:04.135330 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.135299 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:04.165947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.163920 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:04.165947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.165256 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:04.238846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.234466 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" podStartSLOduration=4.72123066 podStartE2EDuration="5.234442691s" totalImagesPullingTime="513.212031ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:24:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:04.203472483 +0000 UTC m=+1826.236568554" watchObservedRunningTime="2026-10-31 00:25:04.234442691 +0000 UTC m=+1826.267538765" Oct 31 00:25:04.350313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:04.350272 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" event={"ID":"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638","Type":"ContainerStarted","Data":"bbd9183929df10cd60481b5192eeba326a9b5a0125457cb5d7ef8e24dd88490f"} Oct 31 00:25:05.128447 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:05.127049 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:05.128447 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:05.127109 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:05.207470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:05.204237 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:05.207470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:05.204293 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:05.218871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:05.218738 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:06.131960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:06.131920 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:06.132273 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:06.131984 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:06.161851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:06.147201 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:06.161851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:06.147263 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:07.127014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.126972 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:25:07.206516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.206480 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:07.206826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.206773 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:07.207472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.207449 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:07.207650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.207629 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:07.364519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.364457 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:07.512055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:07.512015 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:25:08.111921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.111869 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:25:08.112528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.112497 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:08.114342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.114314 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:08.114513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.114491 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:08.156079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.155630 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:08.159978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.159942 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:08.183663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.183614 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:25:08.184256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.184226 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:25:08.337073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.334734 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:25:08.337378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.337362 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:25:08.422231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:08.422070 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:09.179810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:09.179036 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:09.179810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:09.179095 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:09.179810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:09.179481 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:09.179810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:09.179514 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:09.844960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:09.844908 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:10.112068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:10.112031 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" start-of-body= Oct 31 00:25:10.112344 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:10.112312 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.132:15021/healthz/ready\": dial tcp 10.134.0.132:15021: connect: connection refused" Oct 31 00:25:10.169343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:10.169305 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" start-of-body= Oct 31 00:25:10.169649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:10.169616 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.133:15021/healthz/ready\": dial tcp 10.134.0.133:15021: connect: connection refused" Oct 31 00:25:11.116487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.116443 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:11.116992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.116969 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:11.118312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.118292 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:25:11.150745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.150717 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:11.151169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.151140 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:11.152032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.152013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:25:11.850264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:11.850224 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:25:13.254158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:13.254122 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:25:13.254653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:13.254468 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:15.699127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.699089 6234 generic.go:356] "Generic (PLEG): container finished" podID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerID="a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8" exitCode=0 Oct 31 00:25:15.699627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.699161 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerDied","Data":"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8"} Oct 31 00:25:15.699719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.699635 6234 scope.go:122] "RemoveContainer" containerID="a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8" Oct 31 00:25:15.701589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.701557 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:15.701669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.701631 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:15.701705 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:15.701673 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:16.716502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:16.716471 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerStarted","Data":"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4"} Oct 31 00:25:17.162870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:17.162458 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:25:17.163993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:17.163964 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:17.164221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:17.164186 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:18.163822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:18.160192 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:18.163822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:18.160258 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:18.693835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:18.693771 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:25:18.694093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:18.694047 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" containerID="cri-o://b1b4a3e33b67cb4722cb973e1bb866b05387c2d29ce849eb3b6095b8b96d9c6f" gracePeriod=30 Oct 31 00:25:19.157875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:19.157255 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:19.157875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:19.157334 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:20.163812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.161917 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:20.163812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.161986 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:20.983280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.983241 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:25:20.988978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.988952 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:20.997394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.997366 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:25:20.997563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:20.997516 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-7b1fc7\"" Oct 31 00:25:21.061816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.060682 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-7b1fc7-dockercfg-66wkm\"" Oct 31 00:25:21.195809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.192895 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cvttr\" (UniqueName: \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.195809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.193170 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.205623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.205585 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:21.205989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.205955 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:21.297862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.297667 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cvttr\" (UniqueName: \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.298042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.298003 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.321815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.319723 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.416991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.408835 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cvttr\" (UniqueName: \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\") pod \"payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.625914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.625538 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:21.993394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:21.993361 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:25:21.996008 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:25:21.995976 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9370837c_e0ba_4889_8d68_136421253993.slice/crio-68ec0747154b060dc143456ffd1449147f1864b999d0cc4a3b42c1834fc16128 WatchSource:0}: Error finding container 68ec0747154b060dc143456ffd1449147f1864b999d0cc4a3b42c1834fc16128: Status 404 returned error can't find the container with id 68ec0747154b060dc143456ffd1449147f1864b999d0cc4a3b42c1834fc16128 Oct 31 00:25:22.162650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.161248 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:22.162650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.161305 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:22.873731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.873686 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" event={"ID":"9370837c-e0ba-4889-8d68-136421253993","Type":"ContainerStarted","Data":"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1"} Oct 31 00:25:22.874339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.874318 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:22.874471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.874457 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" event={"ID":"9370837c-e0ba-4889-8d68-136421253993","Type":"ContainerStarted","Data":"68ec0747154b060dc143456ffd1449147f1864b999d0cc4a3b42c1834fc16128"} Oct 31 00:25:22.898875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:22.898815 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" podStartSLOduration=2.898799639 podStartE2EDuration="2.898799639s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:22.893362253 +0000 UTC m=+1844.926458324" watchObservedRunningTime="2026-10-31 00:25:22.898799639 +0000 UTC m=+1844.931895706" Oct 31 00:25:23.152870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:23.152212 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body= Oct 31 00:25:23.152870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:23.152307 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" Oct 31 00:25:23.889811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:23.889749 6234 generic.go:356] "Generic (PLEG): container finished" podID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerID="b1b4a3e33b67cb4722cb973e1bb866b05387c2d29ce849eb3b6095b8b96d9c6f" exitCode=0 Oct 31 00:25:23.890253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:23.890057 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" event={"ID":"506e0f41-df4b-43ab-8e7c-7e3c33378153","Type":"ContainerDied","Data":"b1b4a3e33b67cb4722cb973e1bb866b05387c2d29ce849eb3b6095b8b96d9c6f"} Oct 31 00:25:24.122814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.119990 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:25:24.153054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.153027 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:25:24.153556 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.153495 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:25:24.154536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.154513 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:25:24.162747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.162721 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:25:24.163818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.163769 6234 cpu_manager.go:414] "containerMap: removing container" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" Oct 31 00:25:24.163818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.163816 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" Oct 31 00:25:24.163972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.163961 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" containerName="istio-proxy" Oct 31 00:25:24.168083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.168065 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.191462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.190356 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:25:24.222389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222295 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222350 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222445 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222631 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222679 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.222750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222694 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222836 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.222944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.222919 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.223184 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223164 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.223275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223245 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\") pod \"506e0f41-df4b-43ab-8e7c-7e3c33378153\" (UID: \"506e0f41-df4b-43ab-8e7c-7e3c33378153\") " Oct 31 00:25:24.223336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223273 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.223641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223613 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:25:24.223758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223634 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.223845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223816 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.223938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223918 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224048 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224105 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224168 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224199 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-848b8\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224424 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224501 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.224632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224614 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.224696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224638 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/506e0f41-df4b-43ab-8e7c-7e3c33378153-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.224696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.224653 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.224909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.223921 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.225910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.225678 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.228542 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.228466 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:24.228873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.228848 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "kube-api-access-fs4jc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:24.244900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.244861 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:25:24.244900 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.244869 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo" pod "506e0f41-df4b-43ab-8e7c-7e3c33378153" (UID: "506e0f41-df4b-43ab-8e7c-7e3c33378153"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.325902 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326050 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326120 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326424 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326485 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326536 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326573 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-848b8\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326707 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.326757 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.333755 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.338577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.333833 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.342417 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.342713 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.344809 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345169 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345195 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345211 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345233 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fs4jc\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-kube-api-access-fs4jc\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345253 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.345272 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/506e0f41-df4b-43ab-8e7c-7e3c33378153-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.346436 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.351990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.349075 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.356432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.356410 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.356534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.356471 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-848b8\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.507166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.507079 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:24.805511 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:25:24.805478 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb3faba1_cd88_464b_b4c9_3aa583cc0c44.slice/crio-cdca0e4fd0772d29e8f39e6d97b5fef0f5fbc9bfed8b5b2d8e40565cefe6f8cc WatchSource:0}: Error finding container cdca0e4fd0772d29e8f39e6d97b5fef0f5fbc9bfed8b5b2d8e40565cefe6f8cc: Status 404 returned error can't find the container with id cdca0e4fd0772d29e8f39e6d97b5fef0f5fbc9bfed8b5b2d8e40565cefe6f8cc Oct 31 00:25:24.807325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.807299 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:25:24.809678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.809639 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:24.809767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.809727 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:24.809861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.809772 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:25:24.902221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.902184 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" event={"ID":"cb3faba1-cd88-464b-b4c9-3aa583cc0c44","Type":"ContainerStarted","Data":"cdca0e4fd0772d29e8f39e6d97b5fef0f5fbc9bfed8b5b2d8e40565cefe6f8cc"} Oct 31 00:25:24.904653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.904624 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" event={"ID":"506e0f41-df4b-43ab-8e7c-7e3c33378153","Type":"ContainerDied","Data":"6d5465a30a12d02b633e2b321cc5ef4610ba5275ef77b53907ef17862fd89d2a"} Oct 31 00:25:24.904829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.904687 6234 scope.go:122] "RemoveContainer" containerID="b1b4a3e33b67cb4722cb973e1bb866b05387c2d29ce849eb3b6095b8b96d9c6f" Oct 31 00:25:24.904829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.904633 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f" Oct 31 00:25:24.976279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.976238 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:25:24.984329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:24.984291 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-m2s5f"] Oct 31 00:25:25.914306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:25.914270 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" event={"ID":"cb3faba1-cd88-464b-b4c9-3aa583cc0c44","Type":"ContainerStarted","Data":"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff"} Oct 31 00:25:26.048816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:26.013496 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podStartSLOduration=16.01347522 podStartE2EDuration="16.01347522s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:25.95627988 +0000 UTC m=+1847.989375977" watchObservedRunningTime="2026-10-31 00:25:26.01347522 +0000 UTC m=+1848.046571291" Oct 31 00:25:26.169123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:26.169021 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="506e0f41-df4b-43ab-8e7c-7e3c33378153" path="/var/lib/kubelet/pods/506e0f41-df4b-43ab-8e7c-7e3c33378153/volumes" Oct 31 00:25:26.507345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:26.507315 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:26.509060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:26.509035 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:26.509124 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:26.509085 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:27.518830 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:27.518356 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:27.518830 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:27.518418 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:28.508091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:28.508039 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:28.508467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:28.508426 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:29.585963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:29.585919 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:29.594973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:29.594929 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:30.527461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:30.527426 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:30.527767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:30.527722 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:31.608809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.600889 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": context deadline exceeded" Oct 31 00:25:31.668808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.643161 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:31.669132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.669087 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:31.770806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.766255 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:31.871863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.856020 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:31.871863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:31.856292 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.025631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.025583 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.056344 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.056293 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.057276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.057244 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:25:32.179937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.155625 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:25:32.200811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.196953 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.258929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.258866 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.513842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.512839 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:32.513842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.512912 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:32.767693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.767594 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": context deadline exceeded" Oct 31 00:25:32.785872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.774913 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:25:32.884809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.883035 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:32.884809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:32.883558 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:25:33.123307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.123183 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": context deadline exceeded" Oct 31 00:25:33.125053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.125011 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:25:33.164601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.164558 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": context deadline exceeded" Oct 31 00:25:33.172036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.171994 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded" Oct 31 00:25:33.181092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.181040 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:25:33.200092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.200046 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:25:33.268973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.220892 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:25:33.268973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.254582 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:25:33.268973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.256235 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:25:33.307007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.276541 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.307007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.277673 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:25:33.307007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.287122 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.307007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.290838 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:25:33.525978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.525931 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" start-of-body= Oct 31 00:25:33.699581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.693959 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.135:15021/healthz/ready\": dial tcp 10.134.0.135:15021: connect: connection refused" Oct 31 00:25:33.699933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.540870 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.780183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.780043 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.884933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.884166 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:33.910814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:33.909838 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:25:33.912204 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:25:33.912145 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:25:34.013818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.013540 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:25:34.116476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.116386 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:25:34.287808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.286913 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.449032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.448970 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:25:34.467632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.462117 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" probeResult="failure" output="Get \"https://10.134.0.57:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.660128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.660083 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded" Oct 31 00:25:34.793148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.792988 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.986449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.985550 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:25:34.986449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.985651 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:34.986449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.985700 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:25:34.986449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.986064 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:34.993821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.991349 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:34.993821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:34.992272 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:25:35.010862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:35.006919 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:25:35.012626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:35.012599 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:25:35.250143 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:35.250113 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:25:35.280530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:35.278624 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:25:35.318602 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:35.318570 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:25:43.757144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:43.757106 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11085-e4ca10"] Oct 31 00:25:43.822655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:43.822621 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11085-e4ca10"] Oct 31 00:25:43.822982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:43.822933 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:43.830886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:43.830668 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lr8lt\"" Oct 31 00:25:43.950482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:43.950448 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s57c4\" (UniqueName: \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\") pod \"test-curl-11085-e4ca10\" (UID: \"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8\") " pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:44.053352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:44.053262 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s57c4\" (UniqueName: \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\") pod \"test-curl-11085-e4ca10\" (UID: \"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8\") " pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:44.066508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:44.066480 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s57c4\" (UniqueName: \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\") pod \"test-curl-11085-e4ca10\" (UID: \"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8\") " pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:44.144093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:44.144064 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:44.430102 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:25:44.430060 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84b5d453_6f4d_4f2d_ae85_93c722b3a9c8.slice/crio-e40ee8086d774881066a6b2289eb853cac22e1b725fae2a1757b1e0ee6c461a3 WatchSource:0}: Error finding container e40ee8086d774881066a6b2289eb853cac22e1b725fae2a1757b1e0ee6c461a3: Status 404 returned error can't find the container with id e40ee8086d774881066a6b2289eb853cac22e1b725fae2a1757b1e0ee6c461a3 Oct 31 00:25:44.432762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:44.432737 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11085-e4ca10"] Oct 31 00:25:45.249410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:45.249373 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11085-e4ca10" event={"ID":"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8","Type":"ContainerStarted","Data":"954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0"} Oct 31 00:25:45.249410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:45.249418 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11085-e4ca10" event={"ID":"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8","Type":"ContainerStarted","Data":"e40ee8086d774881066a6b2289eb853cac22e1b725fae2a1757b1e0ee6c461a3"} Oct 31 00:25:45.276683 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:45.276631 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11085-e4ca10" podStartSLOduration=2.276618532 podStartE2EDuration="2.276618532s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:25:45.271744318 +0000 UTC m=+1867.304840389" watchObservedRunningTime="2026-10-31 00:25:45.276618532 +0000 UTC m=+1867.309714602" Oct 31 00:25:46.020931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:46.020894 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11085-e4ca10"] Oct 31 00:25:46.027252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:46.027225 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11085-e4ca10"] Oct 31 00:25:47.265739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:47.265697 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11085-e4ca10" podUID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" containerName="test-curl-11085-e4ca10" containerID="cri-o://954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0" gracePeriod=2 Oct 31 00:25:49.501188 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:49.501154 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:49.639270 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:49.639191 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\" (UniqueName: \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\") pod \"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8\" (UID: \"84b5d453-6f4d-4f2d-ae85-93c722b3a9c8\") " Oct 31 00:25:49.641491 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:49.641448 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4" pod "84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" (UID: "84b5d453-6f4d-4f2d-ae85-93c722b3a9c8"). InnerVolumeSpecName "kube-api-access-s57c4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:25:49.752500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:49.743464 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s57c4\" (UniqueName: \"kubernetes.io/projected/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8-kube-api-access-s57c4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:25:50.086430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.086393 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" path="/var/lib/kubelet/pods/84b5d453-6f4d-4f2d-ae85-93c722b3a9c8/volumes" Oct 31 00:25:50.308587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.308548 6234 generic.go:356] "Generic (PLEG): container finished" podID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" containerID="954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0" exitCode=137 Oct 31 00:25:50.308803 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.308595 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-e4ca10" Oct 31 00:25:50.308803 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.308650 6234 scope.go:122] "RemoveContainer" containerID="954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0" Oct 31 00:25:50.320659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.320633 6234 scope.go:122] "RemoveContainer" containerID="954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0" Oct 31 00:25:50.320995 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:25:50.320974 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0\": container with ID starting with 954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0 not found: ID does not exist" containerID="954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0" Oct 31 00:25:50.321075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:50.321006 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0"} err="failed to get container status \"954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0\": rpc error: code = NotFound desc = could not find container \"954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0\": container with ID starting with 954f47a497b8fe93d6acd4bab79efd3920a00fe79fd85b33ca396a667b7913c0 not found: ID does not exist" Oct 31 00:25:56.848178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:25:56.848137 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:26:03.170831 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:03.170047 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:26:07.188706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:07.188664 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:26:07.204627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:07.204594 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:26:11.871864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:11.871821 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" containerID="cri-o://5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68" gracePeriod=45 Oct 31 00:26:12.188607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.188572 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:26:12.189353 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.189325 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" containerID="cri-o://57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c" gracePeriod=25 Oct 31 00:26:12.205315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.205283 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" containerID="cri-o://58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381" gracePeriod=25 Oct 31 00:26:12.251017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.250981 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\" (UniqueName: \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251260 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\" (UniqueName: \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251350 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251525 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251744 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251877 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\") pod \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\" (UID: \"8769c5e0-af3a-4da6-ae9e-f316e967ce48\") " Oct 31 00:26:12.251970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.251920 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.252173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.252141 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.252383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.252363 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.253063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.253035 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.253181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.253068 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.253181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.253085 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.253758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.253720 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:26:12.254452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.254420 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:12.255406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.255370 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz" pod "8769c5e0-af3a-4da6-ae9e-f316e967ce48" (UID: "8769c5e0-af3a-4da6-ae9e-f316e967ce48"). InnerVolumeSpecName "kube-api-access-745zz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:12.355099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.355067 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8769c5e0-af3a-4da6-ae9e-f316e967ce48-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.355099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.355095 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-745zz\" (UniqueName: \"kubernetes.io/projected/8769c5e0-af3a-4da6-ae9e-f316e967ce48-kube-api-access-745zz\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.355099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.355106 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8769c5e0-af3a-4da6-ae9e-f316e967ce48-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:12.751898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.750816 6234 generic.go:356] "Generic (PLEG): container finished" podID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerID="5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68" exitCode=0 Oct 31 00:26:12.751898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.751170 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" event={"ID":"8769c5e0-af3a-4da6-ae9e-f316e967ce48","Type":"ContainerDied","Data":"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68"} Oct 31 00:26:12.751898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.751233 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" event={"ID":"8769c5e0-af3a-4da6-ae9e-f316e967ce48","Type":"ContainerDied","Data":"3e05ed2d239838aa47876e3365dde9ebe59998a5d6ccc5a2a9ff218011dd708d"} Oct 31 00:26:12.751898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.751259 6234 scope.go:122] "RemoveContainer" containerID="5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68" Oct 31 00:26:12.751898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.751554 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k" Oct 31 00:26:12.768508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.768116 6234 scope.go:122] "RemoveContainer" containerID="5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68" Oct 31 00:26:12.768888 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:12.768863 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68\": container with ID starting with 5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68 not found: ID does not exist" containerID="5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68" Oct 31 00:26:12.768981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.768903 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68"} err="failed to get container status \"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68\": rpc error: code = NotFound desc = could not find container \"5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68\": container with ID starting with 5c64144b403d43f558d2cc9c25e12716eeb1710c2a66c8f30a31a4b1342c9b68 not found: ID does not exist" Oct 31 00:26:12.788619 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.788590 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:26:12.793059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:12.793038 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-19adb9/simulator-w3-19adb9-kserve-5d557b7984-6rd2k"] Oct 31 00:26:14.085189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:14.085145 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" path="/var/lib/kubelet/pods/8769c5e0-af3a-4da6-ae9e-f316e967ce48/volumes" Oct 31 00:26:18.911293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:18.911209 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:26:18.911741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:18.911405 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" containerID="cri-o://8b1836f2b728038dfffda3e1bbbd9474f3a3fbc91d7390cae532736cb42ca0b9" gracePeriod=30 Oct 31 00:26:21.726018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:21.725981 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:26:21.726752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:21.726729 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:26:24.179968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.179935 6234 generic.go:356] "Generic (PLEG): container finished" podID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerID="8b1836f2b728038dfffda3e1bbbd9474f3a3fbc91d7390cae532736cb42ca0b9" exitCode=0 Oct 31 00:26:24.180411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.179989 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" event={"ID":"58ea5482-2d1b-4bd7-9f50-103331af8349","Type":"ContainerDied","Data":"8b1836f2b728038dfffda3e1bbbd9474f3a3fbc91d7390cae532736cb42ca0b9"} Oct 31 00:26:24.218405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.218375 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:26:24.267514 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.267479 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:26:24.268482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268462 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" Oct 31 00:26:24.268557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268486 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" Oct 31 00:26:24.268557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268511 6234 cpu_manager.go:414] "containerMap: removing container" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" Oct 31 00:26:24.268557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268520 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" Oct 31 00:26:24.268676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268575 6234 cpu_manager.go:414] "containerMap: removing container" podUID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" containerName="test-curl-11085-e4ca10" Oct 31 00:26:24.268676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268582 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" containerName="test-curl-11085-e4ca10" Oct 31 00:26:24.268810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268798 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8769c5e0-af3a-4da6-ae9e-f316e967ce48" containerName="main" Oct 31 00:26:24.268849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268820 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" containerName="istio-proxy" Oct 31 00:26:24.268849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.268842 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="84b5d453-6f4d-4f2d-ae85-93c722b3a9c8" containerName="test-curl-11085-e4ca10" Oct 31 00:26:24.273626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.273602 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.290094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.288262 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-d609e2-openshift-default-dockercfg-jlhdn\"" Oct 31 00:26:24.290094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.290050 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:26:24.324696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.324664 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.324866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.324758 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.324985 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325114 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325167 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325214 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325336 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325394 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.325524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.325443 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\") pod \"58ea5482-2d1b-4bd7-9f50-103331af8349\" (UID: \"58ea5482-2d1b-4bd7-9f50-103331af8349\") " Oct 31 00:26:24.326094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.326066 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.326749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.326554 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.326749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.326681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.326967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.326768 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.327053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.326981 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.327328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.327280 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:24.327432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.327339 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.327840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.327816 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:24.329277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.329253 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "kube-api-access-dg678". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:24.329534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.329504 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy" pod "58ea5482-2d1b-4bd7-9f50-103331af8349" (UID: "58ea5482-2d1b-4bd7-9f50-103331af8349"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431563 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431655 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431693 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431721 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431846 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.431949 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxcwf\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432006 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432056 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432214 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432408 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432429 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432445 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432458 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432471 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/58ea5482-2d1b-4bd7-9f50-103331af8349-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432484 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432498 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/58ea5482-2d1b-4bd7-9f50-103331af8349-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.435720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.432519 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg678\" (UniqueName: \"kubernetes.io/projected/58ea5482-2d1b-4bd7-9f50-103331af8349-kube-api-access-dg678\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:24.533791 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.533754 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.533832 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.533866 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534009 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.533976 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534086 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxcwf\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534149 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534192 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534297 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534355 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534496 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.534768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.534735 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.535030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.535006 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.535109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.535030 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.535215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.535191 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.536435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.536414 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.537424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.537403 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.558955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.558926 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxcwf\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.559802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.559757 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\") pod \"e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:24.602521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:24.602200 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:25.086551 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:25.086477 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ac6ba56_d1b7_44fc_b8ce_4125405a30a9.slice/crio-e4fe8461eaa61da464b3450656b0f22e215b0d2d8267d872431625ad92e10448 WatchSource:0}: Error finding container e4fe8461eaa61da464b3450656b0f22e215b0d2d8267d872431625ad92e10448: Status 404 returned error can't find the container with id e4fe8461eaa61da464b3450656b0f22e215b0d2d8267d872431625ad92e10448 Oct 31 00:26:25.089580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.089543 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:25.089700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.089631 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:25.089700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.089683 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:25.090435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.090417 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:26:25.191010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.190983 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" Oct 31 00:26:25.191484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.190979 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc" event={"ID":"58ea5482-2d1b-4bd7-9f50-103331af8349","Type":"ContainerDied","Data":"fcc0bf27887f096fd7d72d471e0409a3837708731e4aca0e9a848fdd1d090c66"} Oct 31 00:26:25.191484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.191119 6234 scope.go:122] "RemoveContainer" containerID="8b1836f2b728038dfffda3e1bbbd9474f3a3fbc91d7390cae532736cb42ca0b9" Oct 31 00:26:25.193036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.193008 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerStarted","Data":"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1"} Oct 31 00:26:25.193203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.193061 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerStarted","Data":"e4fe8461eaa61da464b3450656b0f22e215b0d2d8267d872431625ad92e10448"} Oct 31 00:26:25.227745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.227683 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podStartSLOduration=5.22766913 podStartE2EDuration="5.22766913s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:25.222749707 +0000 UTC m=+1907.255845781" watchObservedRunningTime="2026-10-31 00:26:25.22766913 +0000 UTC m=+1907.260765200" Oct 31 00:26:25.243926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.243898 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:26:25.250459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.250432 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-pw2bc"] Oct 31 00:26:25.603108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:25.603072 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:26.085560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:26.085528 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="58ea5482-2d1b-4bd7-9f50-103331af8349" path="/var/lib/kubelet/pods/58ea5482-2d1b-4bd7-9f50-103331af8349/volumes" Oct 31 00:26:26.608589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:26.608561 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:27.212894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:27.212858 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:27.213901 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:27.213859 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:26:29.252575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.252541 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:26:29.257590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.257569 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.268022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.268002 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:26:29.268513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.268489 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-d609e2\"" Oct 31 00:26:29.504054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.503863 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.505017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.504716 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppd9v\" (UniqueName: \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.614525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.614474 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.614721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.614677 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppd9v\" (UniqueName: \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.616011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.615982 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.684808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.678128 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppd9v\" (UniqueName: \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\") pod \"payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:29.877569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:29.877484 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:30.428255 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:30.428225 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaa8b4f3c_701d_4d1c_aa78_827fa3d28060.slice/crio-8f337e4b7e79e3cf7bbfd9dd5cc138c9d4b35d4ff8ac072f61872ac33ec0c741 WatchSource:0}: Error finding container 8f337e4b7e79e3cf7bbfd9dd5cc138c9d4b35d4ff8ac072f61872ac33ec0c741: Status 404 returned error can't find the container with id 8f337e4b7e79e3cf7bbfd9dd5cc138c9d4b35d4ff8ac072f61872ac33ec0c741 Oct 31 00:26:30.428941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:30.428856 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:26:31.294808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:31.289817 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" event={"ID":"aa8b4f3c-701d-4d1c-aa78-827fa3d28060","Type":"ContainerStarted","Data":"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0"} Oct 31 00:26:31.294808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:31.289891 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" event={"ID":"aa8b4f3c-701d-4d1c-aa78-827fa3d28060","Type":"ContainerStarted","Data":"8f337e4b7e79e3cf7bbfd9dd5cc138c9d4b35d4ff8ac072f61872ac33ec0c741"} Oct 31 00:26:31.294808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:31.289919 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:31.735809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:31.735020 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:26:31.735809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:31.735549 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:26:32.560812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.557222 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:26:32.614879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.614572 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" podStartSLOduration=3.6145397580000003 podStartE2EDuration="3.614539758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:31.342203564 +0000 UTC m=+1913.375299652" watchObservedRunningTime="2026-10-31 00:26:32.614539758 +0000 UTC m=+1914.647635829" Oct 31 00:26:32.627682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.627055 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\" (UniqueName: \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\") pod \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " Oct 31 00:26:32.627682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.627119 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\") pod \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\" (UID: \"ebb30701-65ba-4746-bd1f-adc2f4a8acd6\") " Oct 31 00:26:32.668251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.656591 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume" pod "ebb30701-65ba-4746-bd1f-adc2f4a8acd6" (UID: "ebb30701-65ba-4746-bd1f-adc2f4a8acd6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:32.668251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.663486 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p" pod "ebb30701-65ba-4746-bd1f-adc2f4a8acd6" (UID: "ebb30701-65ba-4746-bd1f-adc2f4a8acd6"). InnerVolumeSpecName "kube-api-access-qq52p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:32.668251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.666769 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:26:32.668528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.668498 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" Oct 31 00:26:32.668609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.668528 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" Oct 31 00:26:32.671867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.668914 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerName="payload-processing" Oct 31 00:26:32.672307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.672283 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qq52p\" (UniqueName: \"kubernetes.io/projected/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-kube-api-access-qq52p\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:32.673899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.673876 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ebb30701-65ba-4746-bd1f-adc2f4a8acd6-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:32.678926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.678904 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.679722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.678955 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:26:32.709538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.709510 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:26:32.759206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.759123 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-d609e2-dockercfg-wjs7s\"" Oct 31 00:26:32.854825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.849658 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.854825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.849772 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tdfcs\" (UniqueName: \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.951045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.951014 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\" (UniqueName: \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\") pod \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " Oct 31 00:26:32.951215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.951130 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\") pod \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\" (UID: \"a8ac39d7-28c4-48eb-a612-506a2d1df18d\") " Oct 31 00:26:32.951522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.951492 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume" pod "a8ac39d7-28c4-48eb-a612-506a2d1df18d" (UID: "a8ac39d7-28c4-48eb-a612-506a2d1df18d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:32.951767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.951713 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.951767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.951797 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tdfcs\" (UniqueName: \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.952021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.952003 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a8ac39d7-28c4-48eb-a612-506a2d1df18d-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:32.952465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.952442 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:32.953395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.953373 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8" pod "a8ac39d7-28c4-48eb-a612-506a2d1df18d" (UID: "a8ac39d7-28c4-48eb-a612-506a2d1df18d"). InnerVolumeSpecName "kube-api-access-nxzt8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:32.965602 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:32.965579 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tdfcs\" (UniqueName: \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\") pod \"payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:33.008539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.008510 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:33.053719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.053684 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxzt8\" (UniqueName: \"kubernetes.io/projected/a8ac39d7-28c4-48eb-a612-506a2d1df18d-kube-api-access-nxzt8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:33.167422 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:33.167387 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:26:33.304721 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:33.304687 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc92ce56_f849_4ea0_8aaf_b28ed7e244b3.slice/crio-a450a204b577abf3614b6dd19c6fb3bb115c9792cecdaccd1190da0dbdc1a4b3 WatchSource:0}: Error finding container a450a204b577abf3614b6dd19c6fb3bb115c9792cecdaccd1190da0dbdc1a4b3: Status 404 returned error can't find the container with id a450a204b577abf3614b6dd19c6fb3bb115c9792cecdaccd1190da0dbdc1a4b3 Oct 31 00:26:33.307460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.306985 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:26:33.317261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.317236 6234 generic.go:356] "Generic (PLEG): container finished" podID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerID="58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381" exitCode=0 Oct 31 00:26:33.317362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.317298 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" Oct 31 00:26:33.317362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.317325 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" event={"ID":"a8ac39d7-28c4-48eb-a612-506a2d1df18d","Type":"ContainerDied","Data":"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381"} Oct 31 00:26:33.317489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.317390 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt" event={"ID":"a8ac39d7-28c4-48eb-a612-506a2d1df18d","Type":"ContainerDied","Data":"8fc09ed24d18f748cb6c9cc68c2d1401e30fe95e295fa191b8251904e57e8305"} Oct 31 00:26:33.317489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.317426 6234 scope.go:122] "RemoveContainer" containerID="58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381" Oct 31 00:26:33.320059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.319670 6234 generic.go:356] "Generic (PLEG): container finished" podID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" containerID="57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c" exitCode=0 Oct 31 00:26:33.320059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.319935 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" event={"ID":"ebb30701-65ba-4746-bd1f-adc2f4a8acd6","Type":"ContainerDied","Data":"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c"} Oct 31 00:26:33.320059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.319986 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" event={"ID":"ebb30701-65ba-4746-bd1f-adc2f4a8acd6","Type":"ContainerDied","Data":"b304850300b74ca0d38fe462a58f8c93e3f77c203711d73dcd03d810bab9b4ab"} Oct 31 00:26:33.320243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.320063 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms" Oct 31 00:26:33.331474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.331454 6234 scope.go:122] "RemoveContainer" containerID="58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381" Oct 31 00:26:33.331833 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:33.331806 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381\": container with ID starting with 58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381 not found: ID does not exist" containerID="58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381" Oct 31 00:26:33.331930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.331842 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381"} err="failed to get container status \"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381\": rpc error: code = NotFound desc = could not find container \"58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381\": container with ID starting with 58428e161a48991e9907781fda71bf7283afedbffe9b3706c485b37c8263d381 not found: ID does not exist" Oct 31 00:26:33.331930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.331863 6234 scope.go:122] "RemoveContainer" containerID="57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c" Oct 31 00:26:33.348058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.347951 6234 scope.go:122] "RemoveContainer" containerID="57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c" Oct 31 00:26:33.348360 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:33.348333 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c\": container with ID starting with 57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c not found: ID does not exist" containerID="57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c" Oct 31 00:26:33.348453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.348367 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c"} err="failed to get container status \"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c\": rpc error: code = NotFound desc = could not find container \"57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c\": container with ID starting with 57f6b494512fb18cfadf73831d5f5acc3a95713ca44b4597adc12d31ff3e1d9c not found: ID does not exist" Oct 31 00:26:33.368298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.368266 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:26:33.373054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.373031 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-08673d2c-77fbdd7d57-jwnms"] Oct 31 00:26:33.382353 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.382328 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:26:33.389099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:33.389076 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-5sxnt"] Oct 31 00:26:34.113959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.113924 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" path="/var/lib/kubelet/pods/a8ac39d7-28c4-48eb-a612-506a2d1df18d/volumes" Oct 31 00:26:34.114749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.114725 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ebb30701-65ba-4746-bd1f-adc2f4a8acd6" path="/var/lib/kubelet/pods/ebb30701-65ba-4746-bd1f-adc2f4a8acd6/volumes" Oct 31 00:26:34.337994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.337920 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" event={"ID":"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3","Type":"ContainerStarted","Data":"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b"} Oct 31 00:26:34.338187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.338009 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" event={"ID":"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3","Type":"ContainerStarted","Data":"a450a204b577abf3614b6dd19c6fb3bb115c9792cecdaccd1190da0dbdc1a4b3"} Oct 31 00:26:34.338187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.338043 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:34.380388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:34.380256 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" podStartSLOduration=5.3801293900000005 podStartE2EDuration="5.38012939s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:34.377010832 +0000 UTC m=+1916.410106902" watchObservedRunningTime="2026-10-31 00:26:34.38012939 +0000 UTC m=+1916.413225460" Oct 31 00:26:40.298008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.297981 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:26:40.298399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.298239 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" containerID="cri-o://87db2d0dc6c18319f1337697452d99e498a779f578639f74076aa1308fa3194a" gracePeriod=30 Oct 31 00:26:40.310284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.310268 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:26:40.310628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.310599 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" containerID="cri-o://7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c" gracePeriod=30 Oct 31 00:26:40.319961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.319929 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:26:40.320230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.320200 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" containerID="cri-o://6fd39e8a89ed5b6e7c76cb3a664ed7c5fb0fc9ae8d2302b4050f3a7a50600472" gracePeriod=30 Oct 31 00:26:40.637810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.632270 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:40.637810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:40.632399 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:41.682805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:41.678866 6234 patch_prober.go:29] interesting pod/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:26:41.682805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:41.678971 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:26:42.303982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:42.303941 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:26:45.425693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.425648 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:26:45.753001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.752955 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:26:45.836569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.836526 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866322 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866389 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866442 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866654 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866809 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866937 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.866994 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.867049 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\") pod \"af16b26e-fdaa-491c-9827-22cfd36bf957\" (UID: \"af16b26e-fdaa-491c-9827-22cfd36bf957\") " Oct 31 00:26:45.868846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.868709 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.869043 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.869321 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.869546 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.869605 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.869911 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870431 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870450 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870515 6234 cpu_manager.go:414] "containerMap: removing container" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870524 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870762 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerName="istio-proxy" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.870813 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8ac39d7-28c4-48eb-a612-506a2d1df18d" containerName="payload-pre-processing" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.877104 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.878725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.878460 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:45.880123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.879998 6234 generic.go:356] "Generic (PLEG): container finished" podID="af16b26e-fdaa-491c-9827-22cfd36bf957" containerID="7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c" exitCode=0 Oct 31 00:26:45.880123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.880082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" event={"ID":"af16b26e-fdaa-491c-9827-22cfd36bf957","Type":"ContainerDied","Data":"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c"} Oct 31 00:26:45.880298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.880129 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" event={"ID":"af16b26e-fdaa-491c-9827-22cfd36bf957","Type":"ContainerDied","Data":"13041bc3726912cf8d5a260ee6e42fe1f946fa1c893a6553401f23affc4d9a3b"} Oct 31 00:26:45.880298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.880150 6234 scope.go:122] "RemoveContainer" containerID="7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c" Oct 31 00:26:45.885860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.885827 6234 generic.go:356] "Generic (PLEG): container finished" podID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerID="87db2d0dc6c18319f1337697452d99e498a779f578639f74076aa1308fa3194a" exitCode=0 Oct 31 00:26:45.885973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.885937 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" event={"ID":"e6b0be86-0695-4bfd-a065-e395378e4a66","Type":"ContainerDied","Data":"87db2d0dc6c18319f1337697452d99e498a779f578639f74076aa1308fa3194a"} Oct 31 00:26:45.891316 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.891288 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.891964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.891940 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "kube-api-access-lk55t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:45.920161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.901342 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token" pod "af16b26e-fdaa-491c-9827-22cfd36bf957" (UID: "af16b26e-fdaa-491c-9827-22cfd36bf957"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:45.920161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.915637 6234 generic.go:356] "Generic (PLEG): container finished" podID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerID="6fd39e8a89ed5b6e7c76cb3a664ed7c5fb0fc9ae8d2302b4050f3a7a50600472" exitCode=0 Oct 31 00:26:45.920161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.915695 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" event={"ID":"f531d8ab-90a1-4433-8131-da5ac62a69eb","Type":"ContainerDied","Data":"6fd39e8a89ed5b6e7c76cb3a664ed7c5fb0fc9ae8d2302b4050f3a7a50600472"} Oct 31 00:26:45.924731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.924516 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:26:45.939065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.935647 6234 scope.go:122] "RemoveContainer" containerID="7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c" Oct 31 00:26:45.939065 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:45.938378 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c\": container with ID starting with 7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c not found: ID does not exist" containerID="7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c" Oct 31 00:26:45.939065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.938424 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c"} err="failed to get container status \"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c\": rpc error: code = NotFound desc = could not find container \"7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c\": container with ID starting with 7efccd1182c79b25edffa7920591581bbc73aa203a829be0015d1d6a6743cd5c not found: ID does not exist" Oct 31 00:26:45.945101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.944952 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:26:45.966854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.966824 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:26:45.969710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969563 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.969710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969707 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.969970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969762 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.969970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969891 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.969970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969951 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.970206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.969996 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.970206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970086 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.970206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970091 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.970206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970149 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.970385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970364 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.970565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970547 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\") pod \"e6b0be86-0695-4bfd-a065-e395378e4a66\" (UID: \"e6b0be86-0695-4bfd-a065-e395378e4a66\") " Oct 31 00:26:45.971500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970371 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:45.971624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.970868 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.971717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.971146 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.971842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.971723 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.972039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.972016 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.972341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.972318 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.972548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.972528 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.972723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.972705 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.972918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.972900 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.973147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973123 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.973235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973170 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-twmts\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.973367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973344 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:45.973531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973514 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973628 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973670 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973696 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973712 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973726 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973741 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk55t\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-kube-api-access-lk55t\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973758 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.973775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973771 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973835 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/af16b26e-fdaa-491c-9827-22cfd36bf957-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973851 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973865 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973879 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973892 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e6b0be86-0695-4bfd-a065-e395378e4a66-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.973906 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/af16b26e-fdaa-491c-9827-22cfd36bf957-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:45.974511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.974451 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:45.974707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.974679 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "kube-api-access-sch67". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:45.979919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.979900 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy" pod "e6b0be86-0695-4bfd-a065-e395378e4a66" (UID: "e6b0be86-0695-4bfd-a065-e395378e4a66"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:45.998976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:45.998943 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:26:46.000131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000104 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" Oct 31 00:26:46.000262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000160 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" Oct 31 00:26:46.000262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000187 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" Oct 31 00:26:46.000262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000192 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" Oct 31 00:26:46.000408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000384 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" containerName="istio-proxy" Oct 31 00:26:46.000408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.000396 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" containerName="istio-proxy" Oct 31 00:26:46.004842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.004813 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.015088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.015048 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-719e12-openshift-default-dockercfg-vlkg9\"" Oct 31 00:26:46.030157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.030099 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:26:46.044132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.044098 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:26:46.051978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.051946 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.063164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.062978 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:26:46.074717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.074680 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.074879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.074744 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.074879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.074834 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.074879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.074870 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075076 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075189 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075171 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075374 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075456 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075483 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\") pod \"f531d8ab-90a1-4433-8131-da5ac62a69eb\" (UID: \"f531d8ab-90a1-4433-8131-da5ac62a69eb\") " Oct 31 00:26:46.075892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075838 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.075892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075886 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075911 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf6kn\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075965 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.075995 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076039 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076101 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076137 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076172 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076233 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076298 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076346 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076403 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076429 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076468 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076499 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076557 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076598 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-twmts\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.076846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076634 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b7hxx\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.076846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076670 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.076846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.076726 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.077141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.077115 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.077207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.077129 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.077207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.077139 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.077369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.077346 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.077813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.077772 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:46.078350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.078324 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.078811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.078767 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.079446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079416 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.079615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079594 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.079706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079669 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.079776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079703 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.079776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079768 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.079955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079832 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079961 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079972 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f531d8ab-90a1-4433-8131-da5ac62a69eb-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079984 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.079993 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080001 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080009 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080018 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sch67\" (UniqueName: \"kubernetes.io/projected/e6b0be86-0695-4bfd-a065-e395378e4a66-kube-api-access-sch67\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080028 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e6b0be86-0695-4bfd-a065-e395378e4a66-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080036 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.080297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080184 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.080297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080258 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.081913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080552 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.081913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.080866 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:26:46.081913 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.081718 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:26:46.082216 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.082199 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.082484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.082468 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.082578 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.082469 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54" pod "f531d8ab-90a1-4433-8131-da5ac62a69eb" (UID: "f531d8ab-90a1-4433-8131-da5ac62a69eb"). InnerVolumeSpecName "kube-api-access-s8v54". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:46.085714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.085690 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.102447 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.102416 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-twmts\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.103271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.103247 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\") pod \"e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.181999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.181889 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182024 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182128 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182165 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182192 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf6kn\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182234 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182256 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182290 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182343 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182368 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182407 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182495 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182548 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182583 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182657 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b7hxx\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.182696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182691 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182753 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.182929 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.183078 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.183094 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.183107 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f531d8ab-90a1-4433-8131-da5ac62a69eb-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.183217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.183121 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8v54\" (UniqueName: \"kubernetes.io/projected/f531d8ab-90a1-4433-8131-da5ac62a69eb-kube-api-access-s8v54\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:46.183539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.183498 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.184122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.184096 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.184275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.184164 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.184560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.184538 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.185026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.185002 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.185112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.185077 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.185343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.185319 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.186058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186033 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.186154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186137 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.186331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.186398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186347 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.186501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186484 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.186587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.186569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.187323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.187303 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.195699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.195677 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf6kn\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.202721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.202691 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.202721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.202703 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.203029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.203009 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b7hxx\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\") pod \"e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.243312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.243278 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:46.330363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.330089 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:46.378129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.377148 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:46.545502 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:46.545457 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb84ba267_7a95_4d5c_8657_9433079b95d9.slice/crio-3c61ce3f0ee1a10e31990e01617bab1b753de7efe985bf16c99c234adde08e9a WatchSource:0}: Error finding container 3c61ce3f0ee1a10e31990e01617bab1b753de7efe985bf16c99c234adde08e9a: Status 404 returned error can't find the container with id 3c61ce3f0ee1a10e31990e01617bab1b753de7efe985bf16c99c234adde08e9a Oct 31 00:26:46.545944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.545637 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:26:46.550394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.549739 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.550394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.549858 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.550394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.549910 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.654183 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:46.654152 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod62fc29d0_2c89_4029_baac_7c205319e5f0.slice/crio-0e02192592e98224eb36b07b912d4873d5fee745794d2cf372113c1096e97670 WatchSource:0}: Error finding container 0e02192592e98224eb36b07b912d4873d5fee745794d2cf372113c1096e97670: Status 404 returned error can't find the container with id 0e02192592e98224eb36b07b912d4873d5fee745794d2cf372113c1096e97670 Oct 31 00:26:46.656896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.656864 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:26:46.658187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.658154 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.658308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.658233 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.658308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.658281 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.705834 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:26:46.705542 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod168583c7_7e54_45fa_8911_3627c8c96779.slice/crio-a20f9a17ba5c623b7297c94f71492f97c0a2e5a96fa90fb0bb7fc410e2de11ce WatchSource:0}: Error finding container a20f9a17ba5c623b7297c94f71492f97c0a2e5a96fa90fb0bb7fc410e2de11ce: Status 404 returned error can't find the container with id a20f9a17ba5c623b7297c94f71492f97c0a2e5a96fa90fb0bb7fc410e2de11ce Oct 31 00:26:46.706295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.706270 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:26:46.708931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.708896 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.709026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.708967 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.709026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.709011 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:26:46.932214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.932173 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" event={"ID":"b84ba267-7a95-4d5c-8657-9433079b95d9","Type":"ContainerStarted","Data":"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527"} Oct 31 00:26:46.932498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.932465 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" event={"ID":"b84ba267-7a95-4d5c-8657-9433079b95d9","Type":"ContainerStarted","Data":"3c61ce3f0ee1a10e31990e01617bab1b753de7efe985bf16c99c234adde08e9a"} Oct 31 00:26:46.933873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.933848 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw" Oct 31 00:26:46.936547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.936326 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" event={"ID":"62fc29d0-2c89-4029-baac-7c205319e5f0","Type":"ContainerStarted","Data":"a6fa73229fd3fb44928d460e264ee7921a1b91a5e115aa7766f2531d7daf2ae4"} Oct 31 00:26:46.936547 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.936402 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" event={"ID":"62fc29d0-2c89-4029-baac-7c205319e5f0","Type":"ContainerStarted","Data":"0e02192592e98224eb36b07b912d4873d5fee745794d2cf372113c1096e97670"} Oct 31 00:26:46.939511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.939484 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" event={"ID":"168583c7-7e54-45fa-8911-3627c8c96779","Type":"ContainerStarted","Data":"a442b5e166aa488498a5126ef5475d842f723317f41af0b63e27bee3f0b6c8ce"} Oct 31 00:26:46.939621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:46.939536 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" event={"ID":"168583c7-7e54-45fa-8911-3627c8c96779","Type":"ContainerStarted","Data":"a20f9a17ba5c623b7297c94f71492f97c0a2e5a96fa90fb0bb7fc410e2de11ce"} Oct 31 00:26:47.029200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.011806 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" event={"ID":"e6b0be86-0695-4bfd-a065-e395378e4a66","Type":"ContainerDied","Data":"7bb57eeff5ed21958cb7eccd473b82569f61d1f70dc0a0cddc74234965eba312"} Oct 31 00:26:47.029200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.011886 6234 scope.go:122] "RemoveContainer" containerID="87db2d0dc6c18319f1337697452d99e498a779f578639f74076aa1308fa3194a" Oct 31 00:26:47.029200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.012052 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq" Oct 31 00:26:47.065188 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.065125 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podStartSLOduration=52.065101772 podStartE2EDuration="52.065101772s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:47.038390544 +0000 UTC m=+1929.071486618" watchObservedRunningTime="2026-10-31 00:26:47.065101772 +0000 UTC m=+1929.098197846" Oct 31 00:26:47.121993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.120295 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podStartSLOduration=75.120275942 podStartE2EDuration="1m15.120275942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:47.112700171 +0000 UTC m=+1929.145796242" watchObservedRunningTime="2026-10-31 00:26:47.120275942 +0000 UTC m=+1929.153372013" Oct 31 00:26:47.141664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.141625 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" Oct 31 00:26:47.141895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.141649 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn" event={"ID":"f531d8ab-90a1-4433-8131-da5ac62a69eb","Type":"ContainerDied","Data":"01f810395c6a0f3d7868eecfc7b9d78d001c82fc5656251e0935ad2fe915e01b"} Oct 31 00:26:47.165982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.165951 6234 scope.go:122] "RemoveContainer" containerID="6fd39e8a89ed5b6e7c76cb3a664ed7c5fb0fc9ae8d2302b4050f3a7a50600472" Oct 31 00:26:47.173862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.173834 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:26:47.183982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.183939 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-md2bq"] Oct 31 00:26:47.192223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.192196 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:26:47.199620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.199548 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-sngzw"] Oct 31 00:26:47.221332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.221268 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podStartSLOduration=75.221245528 podStartE2EDuration="1m15.221245528s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:25:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:47.206095599 +0000 UTC m=+1929.239191694" watchObservedRunningTime="2026-10-31 00:26:47.221245528 +0000 UTC m=+1929.254341600" Oct 31 00:26:47.232842 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.232805 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:26:47.243950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.243919 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:47.245088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.245061 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:47.245231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.245116 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:47.246077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.246048 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-19adb9-openshift-default-5fcbcd94f5-dfxrn"] Oct 31 00:26:47.331468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.331374 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:47.333025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.332998 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:47.333160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.333049 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:47.379387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.379355 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:47.380950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.380922 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:47.381054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:47.380970 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:48.312813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.311957 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:48.312813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.312027 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:48.614891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.604136 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:48.614891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.604189 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:48.614891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.612924 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:48.614891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.612984 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:48.850627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.838536 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="af16b26e-fdaa-491c-9827-22cfd36bf957" path="/var/lib/kubelet/pods/af16b26e-fdaa-491c-9827-22cfd36bf957/volumes" Oct 31 00:26:48.920867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.910479 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e6b0be86-0695-4bfd-a065-e395378e4a66" path="/var/lib/kubelet/pods/e6b0be86-0695-4bfd-a065-e395378e4a66/volumes" Oct 31 00:26:48.951098 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.951061 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f531d8ab-90a1-4433-8131-da5ac62a69eb" path="/var/lib/kubelet/pods/f531d8ab-90a1-4433-8131-da5ac62a69eb/volumes" Oct 31 00:26:48.957107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:48.957074 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:26:49.091171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.091125 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:26:49.091581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.091563 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.112702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.112671 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-08673d2c-dockercfg-x9zz8\"" Oct 31 00:26:49.113742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.113136 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-08673d2c\"" Oct 31 00:26:49.213229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.213189 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.213872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.213845 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x7bln\" (UniqueName: \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.306905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.306869 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:49.307254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.307222 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:49.316941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.316904 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.319010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.318286 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.319285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.319266 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x7bln\" (UniqueName: \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.332452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.332421 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:49.332678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.332473 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:49.338205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.338176 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x7bln\" (UniqueName: \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\") pod \"payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:49.413073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.413034 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:49.413347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.413314 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:49.508105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:49.507984 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:50.335596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.335557 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:50.336178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.336143 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:50.368875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.348128 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:50.368875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.348199 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:50.427812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.393692 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:50.429476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.429435 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:50.866824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:50.848578 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:26:51.237375 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.236987 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" event={"ID":"b57f5056-547c-48ce-ac94-7748acca87cc","Type":"ContainerStarted","Data":"b5d402441499c4eefc4b279233b77eab16e3c37571b2d09394f0440116e6b568"} Oct 31 00:26:51.282808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.269932 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:51.282808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.270378 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:51.332927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.332897 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:51.333142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.333116 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:51.390800 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.389884 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:51.390800 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:51.389945 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:52.265146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.264056 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:52.265146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.264118 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:52.331161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.331128 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:52.331392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.331364 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:52.387434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.387396 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:52.387738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.387693 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:52.441824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.413270 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" event={"ID":"b57f5056-547c-48ce-ac94-7748acca87cc","Type":"ContainerStarted","Data":"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68"} Oct 31 00:26:52.441824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.414723 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:26:52.563891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.478498 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:26:52.689213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.657426 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" podStartSLOduration=4.657405283 podStartE2EDuration="4.657405283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:26:52.47834884 +0000 UTC m=+1934.511444911" watchObservedRunningTime="2026-10-31 00:26:52.657405283 +0000 UTC m=+1934.690501355" Oct 31 00:26:52.720164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:52.694177 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:26:53.280233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.280198 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:53.280514 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.280484 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:53.387959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.387906 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:53.409806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.399860 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:53.409806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.392652 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:53.409806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.400304 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:53.908825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:53.900168 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": context deadline exceeded" Oct 31 00:26:54.004508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.004461 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:54.400063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.396818 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body= Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.509173 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.477913 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.438540 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.517693 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.519977 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.520039 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.533102 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:54.570812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.537942 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.60:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:54.823808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.809382 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.58:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:54.925804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.917100 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:26:54.925804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:54.918043 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:26:55.026299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.026266 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:26:55.331549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.331505 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" start-of-body= Oct 31 00:26:55.331870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.331833 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.141:15021/healthz/ready\": dial tcp 10.134.0.141:15021: connect: connection refused" Oct 31 00:26:55.397861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.396933 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" start-of-body= Oct 31 00:26:55.397861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.396984 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.142:15021/healthz/ready\": dial tcp 10.134.0.142:15021: connect: connection refused" Oct 31 00:26:55.756818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.755031 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:55.756818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.756380 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:55.757871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:55.757848 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:26:56.337443 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.337411 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:56.338055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.338036 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:56.338773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.338753 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:26:56.383258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.383227 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:56.383523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.383506 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:56.384275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:56.384256 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:26:57.564665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.564613 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" podUID="653842f3-cb9b-4827-901b-8562e264027f" containerName="payload-processing" containerID="cri-o://70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f" gracePeriod=25 Oct 31 00:26:57.627682 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:57.627647 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653842f3_cb9b_4827_901b_8562e264027f.slice/crio-conmon-70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:57.627682 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:57.627657 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod653842f3_cb9b_4827_901b_8562e264027f.slice/crio-conmon-70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:26:57.721214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.721163 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" podUID="12d742a3-3e30-4cb7-8839-68805e36ed3f" containerName="payload-pre-processing" containerID="cri-o://8521f6179d57f20bf27c6bcb59c02c8a0e28699fb33f860685fdd78ce24a2100" gracePeriod=25 Oct 31 00:26:57.880886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.880857 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:26:57.903224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.902804 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\") pod \"653842f3-cb9b-4827-901b-8562e264027f\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " Oct 31 00:26:57.903224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.902901 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\" (UniqueName: \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\") pod \"653842f3-cb9b-4827-901b-8562e264027f\" (UID: \"653842f3-cb9b-4827-901b-8562e264027f\") " Oct 31 00:26:57.903450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.903280 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume" pod "653842f3-cb9b-4827-901b-8562e264027f" (UID: "653842f3-cb9b-4827-901b-8562e264027f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:57.904403 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.904373 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/653842f3-cb9b-4827-901b-8562e264027f-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:57.907194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.905932 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g" pod "653842f3-cb9b-4827-901b-8562e264027f" (UID: "653842f3-cb9b-4827-901b-8562e264027f"). InnerVolumeSpecName "kube-api-access-8dv8g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:57.926669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.926633 6234 generic.go:356] "Generic (PLEG): container finished" podID="12d742a3-3e30-4cb7-8839-68805e36ed3f" containerID="8521f6179d57f20bf27c6bcb59c02c8a0e28699fb33f860685fdd78ce24a2100" exitCode=0 Oct 31 00:26:57.926887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.926863 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" event={"ID":"12d742a3-3e30-4cb7-8839-68805e36ed3f","Type":"ContainerDied","Data":"8521f6179d57f20bf27c6bcb59c02c8a0e28699fb33f860685fdd78ce24a2100"} Oct 31 00:26:57.928876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.928848 6234 generic.go:356] "Generic (PLEG): container finished" podID="653842f3-cb9b-4827-901b-8562e264027f" containerID="70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f" exitCode=0 Oct 31 00:26:57.930148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.930127 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" Oct 31 00:26:57.930940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.930807 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" event={"ID":"653842f3-cb9b-4827-901b-8562e264027f","Type":"ContainerDied","Data":"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f"} Oct 31 00:26:57.930940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.930864 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5" event={"ID":"653842f3-cb9b-4827-901b-8562e264027f","Type":"ContainerDied","Data":"1bc3a3cce21862c89a54154550c2ed7fefaf188edd104851e004ed5e8c2f73da"} Oct 31 00:26:57.930940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.930896 6234 scope.go:122] "RemoveContainer" containerID="70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f" Oct 31 00:26:57.964261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.964238 6234 scope.go:122] "RemoveContainer" containerID="70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f" Oct 31 00:26:57.964611 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:26:57.964588 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f\": container with ID starting with 70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f not found: ID does not exist" containerID="70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f" Oct 31 00:26:57.964693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.964624 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f"} err="failed to get container status \"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f\": rpc error: code = NotFound desc = could not find container \"70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f\": container with ID starting with 70f0e33fe8bf73f07275013535dc10ef903009601ebc42ddd5c50cd26e0e969f not found: ID does not exist" Oct 31 00:26:57.976421 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.976376 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:26:57.982312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:57.982288 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-34b786fc-78cf48b4d5-wh7b5"] Oct 31 00:26:58.006396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.006368 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dv8g\" (UniqueName: \"kubernetes.io/projected/653842f3-cb9b-4827-901b-8562e264027f-kube-api-access-8dv8g\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:58.021424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.021399 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:26:58.089845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.089745 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="653842f3-cb9b-4827-901b-8562e264027f" path="/var/lib/kubelet/pods/653842f3-cb9b-4827-901b-8562e264027f/volumes" Oct 31 00:26:58.107896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.107865 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\" (UniqueName: \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\") pod \"12d742a3-3e30-4cb7-8839-68805e36ed3f\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " Oct 31 00:26:58.108083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.107906 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\") pod \"12d742a3-3e30-4cb7-8839-68805e36ed3f\" (UID: \"12d742a3-3e30-4cb7-8839-68805e36ed3f\") " Oct 31 00:26:58.108390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.108362 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume" pod "12d742a3-3e30-4cb7-8839-68805e36ed3f" (UID: "12d742a3-3e30-4cb7-8839-68805e36ed3f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:26:58.108988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.108966 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/12d742a3-3e30-4cb7-8839-68805e36ed3f-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:58.110463 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.110439 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl" pod "12d742a3-3e30-4cb7-8839-68805e36ed3f" (UID: "12d742a3-3e30-4cb7-8839-68805e36ed3f"). InnerVolumeSpecName "kube-api-access-qqnjl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:26:58.217851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.213726 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qqnjl\" (UniqueName: \"kubernetes.io/projected/12d742a3-3e30-4cb7-8839-68805e36ed3f-kube-api-access-qqnjl\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:26:58.939217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.939191 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" Oct 31 00:26:58.939687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.939220 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn" event={"ID":"12d742a3-3e30-4cb7-8839-68805e36ed3f","Type":"ContainerDied","Data":"4f4626bd755ddfa4c1c3bb331e01e79bab7243ca26032856120dbf0fb3eaaa51"} Oct 31 00:26:58.939687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.939275 6234 scope.go:122] "RemoveContainer" containerID="8521f6179d57f20bf27c6bcb59c02c8a0e28699fb33f860685fdd78ce24a2100" Oct 31 00:26:58.967632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.967604 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:26:58.972640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:26:58.972619 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-34b786fc-5b45b9cf99-7bbrn"] Oct 31 00:27:00.085136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:00.085101 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12d742a3-3e30-4cb7-8839-68805e36ed3f" path="/var/lib/kubelet/pods/12d742a3-3e30-4cb7-8839-68805e36ed3f/volumes" Oct 31 00:27:01.484620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.484553 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:27:01.485175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.484852 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" containerID="cri-o://2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4" gracePeriod=30 Oct 31 00:27:01.509804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.509752 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:27:01.510047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.510021 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" containerID="cri-o://33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff" gracePeriod=30 Oct 31 00:27:01.516469 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.516442 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:27:01.516649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:01.516628 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" containerID="cri-o://57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527" gracePeriod=30 Oct 31 00:27:03.339492 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:03.339451 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:27:04.887209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:04.887173 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:27:04.992853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:04.992818 6234 patch_prober.go:29] interesting pod/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:27:04.993035 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:04.992894 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:27:06.876060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.876032 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:27:06.879512 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.879487 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:27:06.938327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.937605 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:27:06.939606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939576 6234 cpu_manager.go:414] "containerMap: removing container" podUID="653842f3-cb9b-4827-901b-8562e264027f" containerName="payload-processing" Oct 31 00:27:06.939729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939608 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="653842f3-cb9b-4827-901b-8562e264027f" containerName="payload-processing" Oct 31 00:27:06.939729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939669 6234 cpu_manager.go:414] "containerMap: removing container" podUID="12d742a3-3e30-4cb7-8839-68805e36ed3f" containerName="payload-pre-processing" Oct 31 00:27:06.939729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939678 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12d742a3-3e30-4cb7-8839-68805e36ed3f" containerName="payload-pre-processing" Oct 31 00:27:06.939729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939719 6234 cpu_manager.go:414] "containerMap: removing container" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" Oct 31 00:27:06.939729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939727 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" Oct 31 00:27:06.940044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939742 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.940044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.939749 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.940044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.940009 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="653842f3-cb9b-4827-901b-8562e264027f" containerName="payload-processing" Oct 31 00:27:06.940044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.940041 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="12d742a3-3e30-4cb7-8839-68805e36ed3f" containerName="payload-pre-processing" Oct 31 00:27:06.940236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.940057 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerName="istio-proxy" Oct 31 00:27:06.940236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.940066 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.940236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.940080 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.951135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.951104 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.954894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.954861 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.955023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.954948 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955023 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955011 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.955164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955144 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955210 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955274 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.955609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955433 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.955609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955575 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.955929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955670 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955711 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955757 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.955929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955834 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.956123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955949 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.956123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.955995 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\") pod \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\" (UID: \"cb3faba1-cd88-464b-b4c9-3aa583cc0c44\") " Oct 31 00:27:06.956123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.956056 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.956275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.956134 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\") pod \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\" (UID: \"b4ab648a-8532-4efa-a68d-1748e5d1bcd8\") " Oct 31 00:27:06.961207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.951328 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:06.966773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.966504 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:27:06.970048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.954400 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.960113 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.962190 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.962367 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.962584 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.962955 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.963172 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.970099 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.965705 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.966199 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.967424 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.968239 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.970159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.969869 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "kube-api-access-lfwhw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:06.970749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.970413 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:06.972163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.972137 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:27:06.974684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.974662 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-cc0fe1-openshift-default-dockercfg-c472z\"" Oct 31 00:27:06.978171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.978117 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:06.978276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.978188 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy" pod "b4ab648a-8532-4efa-a68d-1748e5d1bcd8" (UID: "b4ab648a-8532-4efa-a68d-1748e5d1bcd8"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.984482 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.984425 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "kube-api-access-848b8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:06.991726 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.991558 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy" pod "cb3faba1-cd88-464b-b4c9-3aa583cc0c44" (UID: "cb3faba1-cd88-464b-b4c9-3aa583cc0c44"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:06.995275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.995251 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:06.998584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.998562 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.998716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.998702 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerName="istio-proxy" Oct 31 00:27:06.998894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.998880 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" Oct 31 00:27:06.999038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.999025 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" Oct 31 00:27:06.999434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:06.999415 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerName="istio-proxy" Oct 31 00:27:07.005659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.005421 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:07.005659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.005560 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.066331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.065357 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.068346 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069242 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s7h8\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069304 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069336 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069368 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069410 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069462 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.069597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069532 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069644 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069711 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069744 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8rvf\" (UniqueName: \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.069776 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070125 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070175 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070286 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070307 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070303 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070317 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070331 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070345 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070358 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070373 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070388 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lfwhw\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-kube-api-access-lfwhw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070402 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070416 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070430 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070444 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-848b8\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-kube-api-access-848b8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070457 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070473 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070486 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070500 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070515 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b4ab648a-8532-4efa-a68d-1748e5d1bcd8-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.070729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.070529 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb3faba1-cd88-464b-b4c9-3aa583cc0c44-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.123097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.088488 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-08673d2c-serving-cert\"" Oct 31 00:27:07.123097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.096987 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:07.162940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.162386 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:27:07.163254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.162941 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.171937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.171143 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.172155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.171409 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:07.172298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.172272 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.172471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.172453 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.172617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.172601 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.172832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.172809 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:07.172925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.172859 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:07.173097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173075 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.173271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173254 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.173499 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173480 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.173696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173680 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\") pod \"b84ba267-7a95-4d5c-8657-9433079b95d9\" (UID: \"b84ba267-7a95-4d5c-8657-9433079b95d9\") " Oct 31 00:27:07.174083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173489 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:07.174168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.173996 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:07.174377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.174353 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.174523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.174501 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8rvf\" (UniqueName: \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.174662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.174645 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.174871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.174853 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175023 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.175202 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175174 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175249 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.174979 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-d609e2-serving-cert\"" Oct 31 00:27:07.175450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175428 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175670 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s7h8\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175851 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.175986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.175974 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.176066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176057 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.176139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176130 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.176235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176225 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.176339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176328 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.176480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176455 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:07.176570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176554 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.176657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.176167 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.178094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.177112 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.178094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.177527 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.178094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.178018 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:07.178507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.178488 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b84ba267-7a95-4d5c-8657-9433079b95d9-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.178618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.178606 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.178885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.178871 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.179021 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.179008 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.179144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.179131 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.179274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.179260 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.179410 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.179396 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b84ba267-7a95-4d5c-8657-9433079b95d9-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.179682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.178822 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts" pod "b84ba267-7a95-4d5c-8657-9433079b95d9" (UID: "b84ba267-7a95-4d5c-8657-9433079b95d9"). InnerVolumeSpecName "kube-api-access-twmts". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:07.180081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.180058 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.180237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.180216 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.183545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.180865 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.183545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.181001 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.187034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.187006 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8rvf\" (UniqueName: \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\") pod \"maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.188391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.188368 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s7h8\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.194885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.194857 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\") pod \"e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.195238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.195215 6234 generic.go:356] "Generic (PLEG): container finished" podID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" containerID="2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4" exitCode=0 Oct 31 00:27:07.195561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.195546 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" Oct 31 00:27:07.195972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.195941 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerDied","Data":"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4"} Oct 31 00:27:07.196056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.196009 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2" event={"ID":"b4ab648a-8532-4efa-a68d-1748e5d1bcd8","Type":"ContainerDied","Data":"a9315ea50021ce5600b7a1660a24e0dc825ab29fe86dd321f4ff46f2211659a5"} Oct 31 00:27:07.196056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.196040 6234 scope.go:122] "RemoveContainer" containerID="2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4" Oct 31 00:27:07.198326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.198291 6234 generic.go:356] "Generic (PLEG): container finished" podID="b84ba267-7a95-4d5c-8657-9433079b95d9" containerID="57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527" exitCode=0 Oct 31 00:27:07.198505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.198483 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" event={"ID":"b84ba267-7a95-4d5c-8657-9433079b95d9","Type":"ContainerDied","Data":"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527"} Oct 31 00:27:07.198589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.198535 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" event={"ID":"b84ba267-7a95-4d5c-8657-9433079b95d9","Type":"ContainerDied","Data":"3c61ce3f0ee1a10e31990e01617bab1b753de7efe985bf16c99c234adde08e9a"} Oct 31 00:27:07.198675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.198657 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf" Oct 31 00:27:07.200712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.200690 6234 generic.go:356] "Generic (PLEG): container finished" podID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" containerID="33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff" exitCode=0 Oct 31 00:27:07.201503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.200799 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" event={"ID":"cb3faba1-cd88-464b-b4c9-3aa583cc0c44","Type":"ContainerDied","Data":"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff"} Oct 31 00:27:07.201503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.200826 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" event={"ID":"cb3faba1-cd88-464b-b4c9-3aa583cc0c44","Type":"ContainerDied","Data":"cdca0e4fd0772d29e8f39e6d97b5fef0f5fbc9bfed8b5b2d8e40565cefe6f8cc"} Oct 31 00:27:07.201503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.200887 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74" Oct 31 00:27:07.213687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.213599 6234 scope.go:122] "RemoveContainer" containerID="a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8" Oct 31 00:27:07.226795 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.226758 6234 scope.go:122] "RemoveContainer" containerID="2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4" Oct 31 00:27:07.228230 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:07.228210 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4\": container with ID starting with 2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4 not found: ID does not exist" containerID="2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4" Oct 31 00:27:07.228302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.228237 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4"} err="failed to get container status \"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4\": rpc error: code = NotFound desc = could not find container \"2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4\": container with ID starting with 2ab31bddb3f24e097ac4049dc1af3ff5b8f9ca1e4fb9735a4faf16e7f5eb58d4 not found: ID does not exist" Oct 31 00:27:07.228302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.228255 6234 scope.go:122] "RemoveContainer" containerID="a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8" Oct 31 00:27:07.228500 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:07.228482 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8\": container with ID starting with a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8 not found: ID does not exist" containerID="a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8" Oct 31 00:27:07.228537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.228505 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8"} err="failed to get container status \"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8\": rpc error: code = NotFound desc = could not find container \"a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8\": container with ID starting with a48897b1abaec4e8545bbd9f4fe0efab4da6f4bba6c3d6003f9dcb6e9f3621a8 not found: ID does not exist" Oct 31 00:27:07.228537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.228518 6234 scope.go:122] "RemoveContainer" containerID="57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527" Oct 31 00:27:07.238378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.238353 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:27:07.242012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.241988 6234 scope.go:122] "RemoveContainer" containerID="57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527" Oct 31 00:27:07.242305 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:07.242282 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527\": container with ID starting with 57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527 not found: ID does not exist" containerID="57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527" Oct 31 00:27:07.242500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.242316 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527"} err="failed to get container status \"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527\": rpc error: code = NotFound desc = could not find container \"57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527\": container with ID starting with 57e8d72424a2f6ba531d6df01fc4b70bd4cc595ab3d220af711b0590e1752527 not found: ID does not exist" Oct 31 00:27:07.242500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.242339 6234 scope.go:122] "RemoveContainer" containerID="33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff" Oct 31 00:27:07.244883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.244861 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-69w74"] Oct 31 00:27:07.252936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.252912 6234 scope.go:122] "RemoveContainer" containerID="33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff" Oct 31 00:27:07.253370 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:07.253345 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff\": container with ID starting with 33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff not found: ID does not exist" containerID="33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff" Oct 31 00:27:07.253454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.253379 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff"} err="failed to get container status \"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff\": rpc error: code = NotFound desc = could not find container \"33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff\": container with ID starting with 33c982b409db7b43846e9c85fcbf2941466169bfdaa70e8986cc993405379eff not found: ID does not exist" Oct 31 00:27:07.260050 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.260027 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:27:07.265910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.265886 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-qmlc2"] Oct 31 00:27:07.281875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.281837 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-spl8b\" (UniqueName: \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.282099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.282076 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.282222 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.282183 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.282383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.282362 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.282548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.282518 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-twmts\" (UniqueName: \"kubernetes.io/projected/b84ba267-7a95-4d5c-8657-9433079b95d9-kube-api-access-twmts\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.282590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.282551 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b84ba267-7a95-4d5c-8657-9433079b95d9-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:07.290412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.290385 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:27:07.294694 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.294676 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-34b786fc-openshift-default-76cdb5d4f-dszvf"] Oct 31 00:27:07.308839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.308815 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:07.323767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.323715 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:07.385945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.384667 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.385945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.384847 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.385945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.385104 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-spl8b\" (UniqueName: \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.385945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.385311 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.386863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.386603 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.388741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.388698 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.390082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.390055 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.417888 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.417812 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-spl8b\" (UniqueName: \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\") pod \"maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.492502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.492463 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:07.623031 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:07.622995 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92427017_66b5_49af_8c7f_3c8867b057fe.slice/crio-0ec75da9f95436b9ab8158660c3c8be3c21ed9bb920075d695b29a9f083d8202 WatchSource:0}: Error finding container 0ec75da9f95436b9ab8158660c3c8be3c21ed9bb920075d695b29a9f083d8202: Status 404 returned error can't find the container with id 0ec75da9f95436b9ab8158660c3c8be3c21ed9bb920075d695b29a9f083d8202 Oct 31 00:27:07.628909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.626967 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:07.628909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.627049 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:07.628909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.627091 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:07.629198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.629131 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:27:07.635345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.635320 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:07.645118 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:07.644988 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod90907ec4_c51a_4a6b_b798_f7719ccfc821.slice/crio-2bcb1bb3d01e7b1ecc3d1b5eaae121bdb9f2ca6e14851600abafee5823b9f9fa WatchSource:0}: Error finding container 2bcb1bb3d01e7b1ecc3d1b5eaae121bdb9f2ca6e14851600abafee5823b9f9fa: Status 404 returned error can't find the container with id 2bcb1bb3d01e7b1ecc3d1b5eaae121bdb9f2ca6e14851600abafee5823b9f9fa Oct 31 00:27:07.646242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.646225 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:27:07.792436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:07.792395 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:27:07.793208 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:07.793186 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05f6f790_e53a_49c1_8b17_14c82caf5937.slice/crio-c4ceec8e3cbc09559fdc3c22382597a46b8b4e8918ea3c3756284079eb13e3d8 WatchSource:0}: Error finding container c4ceec8e3cbc09559fdc3c22382597a46b8b4e8918ea3c3756284079eb13e3d8: Status 404 returned error can't find the container with id c4ceec8e3cbc09559fdc3c22382597a46b8b4e8918ea3c3756284079eb13e3d8 Oct 31 00:27:08.135315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.135279 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b4ab648a-8532-4efa-a68d-1748e5d1bcd8" path="/var/lib/kubelet/pods/b4ab648a-8532-4efa-a68d-1748e5d1bcd8/volumes" Oct 31 00:27:08.173094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.173012 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b84ba267-7a95-4d5c-8657-9433079b95d9" path="/var/lib/kubelet/pods/b84ba267-7a95-4d5c-8657-9433079b95d9/volumes" Oct 31 00:27:08.174530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.174506 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb3faba1-cd88-464b-b4c9-3aa583cc0c44" path="/var/lib/kubelet/pods/cb3faba1-cd88-464b-b4c9-3aa583cc0c44/volumes" Oct 31 00:27:08.233251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.233214 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" event={"ID":"90907ec4-c51a-4a6b-b798-f7719ccfc821","Type":"ContainerStarted","Data":"2bcb1bb3d01e7b1ecc3d1b5eaae121bdb9f2ca6e14851600abafee5823b9f9fa"} Oct 31 00:27:08.234869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.234842 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerStarted","Data":"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624"} Oct 31 00:27:08.235045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.235030 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerStarted","Data":"0ec75da9f95436b9ab8158660c3c8be3c21ed9bb920075d695b29a9f083d8202"} Oct 31 00:27:08.236480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.236452 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" event={"ID":"05f6f790-e53a-49c1-8b17-14c82caf5937","Type":"ContainerStarted","Data":"c4ceec8e3cbc09559fdc3c22382597a46b8b4e8918ea3c3756284079eb13e3d8"} Oct 31 00:27:08.264628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.264581 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podStartSLOduration=6.264562562 podStartE2EDuration="6.264562562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:08.261063018 +0000 UTC m=+1950.294159106" watchObservedRunningTime="2026-10-31 00:27:08.264562562 +0000 UTC m=+1950.297658634" Oct 31 00:27:08.309874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.309836 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:08.316750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:08.316720 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:09.246796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.246753 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" event={"ID":"90907ec4-c51a-4a6b-b798-f7719ccfc821","Type":"ContainerStarted","Data":"6a1b78c0240e61bc5940add73e430963f97ad9fb760d7151cce6e0236b3ce3b0"} Oct 31 00:27:09.247299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.246837 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:09.248536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.248510 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" event={"ID":"05f6f790-e53a-49c1-8b17-14c82caf5937","Type":"ContainerStarted","Data":"49dd1a506d1b5567cea7edf206447e0d8b15314b3f1d81f4a2de9f1b8acf9cae"} Oct 31 00:27:09.248666 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.248575 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:09.248735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.248724 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:09.249565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.249548 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:27:09.269429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.269386 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" podStartSLOduration=5.787207346 podStartE2EDuration="6.269372159s" totalImagesPullingTime="482.164813ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:09.268657294 +0000 UTC m=+1951.301753364" watchObservedRunningTime="2026-10-31 00:27:09.269372159 +0000 UTC m=+1951.302468229" Oct 31 00:27:09.296129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:09.296085 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" podStartSLOduration=17.817017955 podStartE2EDuration="18.296072396s" totalImagesPullingTime="479.054441ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:26:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:09.292861548 +0000 UTC m=+1951.325957618" watchObservedRunningTime="2026-10-31 00:27:09.296072396 +0000 UTC m=+1951.329168489" Oct 31 00:27:15.263716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:15.263681 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:27:15.264220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:15.264135 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:19.420294 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:19.420261 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:27:24.421354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.421256 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" podUID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" containerName="payload-processing" containerID="cri-o://c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0" gracePeriod=25 Oct 31 00:27:24.727278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.727253 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:27:24.902264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.902224 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\") pod \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " Oct 31 00:27:24.902434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.902425 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\" (UniqueName: \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\") pod \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\" (UID: \"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b\") " Oct 31 00:27:24.902614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.902587 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume" pod "21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" (UID: "21fb8bd9-d844-4fb8-bb6b-8029479d3f4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:24.903234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.903165 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:24.904833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:24.904809 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r" pod "21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" (UID: "21fb8bd9-d844-4fb8-bb6b-8029479d3f4b"). InnerVolumeSpecName "kube-api-access-rwp5r". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.000126 6234 generic.go:356] "Generic (PLEG): container finished" podID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" containerID="c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0" exitCode=0 Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.000234 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.000277 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" event={"ID":"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b","Type":"ContainerDied","Data":"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0"} Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.000360 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28" event={"ID":"21fb8bd9-d844-4fb8-bb6b-8029479d3f4b","Type":"ContainerDied","Data":"147302fa38932c40b8a2b51e7267ad31215d2eef1ce6b0eec7f2468d98601753"} Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.000388 6234 scope.go:122] "RemoveContainer" containerID="c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0" Oct 31 00:27:25.009642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.004776 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwp5r\" (UniqueName: \"kubernetes.io/projected/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b-kube-api-access-rwp5r\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:25.016551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.016527 6234 scope.go:122] "RemoveContainer" containerID="c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0" Oct 31 00:27:25.016966 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:25.016904 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0\": container with ID starting with c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0 not found: ID does not exist" containerID="c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0" Oct 31 00:27:25.016966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.016946 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0"} err="failed to get container status \"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0\": rpc error: code = NotFound desc = could not find container \"c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0\": container with ID starting with c1c51601a72e2ce1daa6aed8a8c2563282b5ba89cc883519d94f00f48ac6a3b0 not found: ID does not exist" Oct 31 00:27:25.046882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.046830 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:27:25.053932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:25.053898 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-719e12-64c9c9475c-jst28"] Oct 31 00:27:26.086390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.086361 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" path="/var/lib/kubelet/pods/21fb8bd9-d844-4fb8-bb6b-8029479d3f4b/volumes" Oct 31 00:27:26.223557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.223521 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:27:26.224415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.224400 6234 cpu_manager.go:414] "containerMap: removing container" podUID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" containerName="payload-processing" Oct 31 00:27:26.224480 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.224418 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" containerName="payload-processing" Oct 31 00:27:26.224588 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.224579 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="21fb8bd9-d844-4fb8-bb6b-8029479d3f4b" containerName="payload-processing" Oct 31 00:27:26.230071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.230049 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.244105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.244076 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d898f3\"" Oct 31 00:27:26.247567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.247544 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:27:26.252580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.252563 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.261829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.261278 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d898f3-dockercfg-56bnk\"" Oct 31 00:27:26.266032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.266009 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:27:26.272633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.272610 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:27:26.338941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.338851 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj5xx\" (UniqueName: \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.339116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.339027 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.454808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.448180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj5xx\" (UniqueName: \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.454808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.448374 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.454808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.448588 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.454808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.448659 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2p5r6\" (UniqueName: \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.454808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.450160 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.503866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.500384 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj5xx\" (UniqueName: \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\") pod \"payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.547805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.546974 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:26.557836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.551394 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.557836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.551470 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2p5r6\" (UniqueName: \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.585500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.585460 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.593286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.593231 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2p5r6\" (UniqueName: \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\") pod \"payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:26.880969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:26.879048 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:27.473164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:27.465604 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:27:27.473647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:27.473500 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:27:27.480736 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:27.480700 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50ff4ab8_21fa_415c_a775_f56c3fca9112.slice/crio-b871f8006af85395702500f1914fa171fb77cb540db98be4ceeb80bb360ede64 WatchSource:0}: Error finding container b871f8006af85395702500f1914fa171fb77cb540db98be4ceeb80bb360ede64: Status 404 returned error can't find the container with id b871f8006af85395702500f1914fa171fb77cb540db98be4ceeb80bb360ede64 Oct 31 00:27:28.044187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.044137 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" event={"ID":"6fa3b215-6d76-4d82-8f57-f23484ecf880","Type":"ContainerStarted","Data":"0cc5fcc1de68fddc1c0490080b47023879a82ddfd14b8ce5803981e44431a70b"} Oct 31 00:27:28.044369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.044204 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" event={"ID":"6fa3b215-6d76-4d82-8f57-f23484ecf880","Type":"ContainerStarted","Data":"8f57e569aa1d8f8036c214c08e021eca96b7e11e22f5215e8dae77f16e7b3f49"} Oct 31 00:27:28.044369 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.044230 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:28.045986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.045960 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" event={"ID":"50ff4ab8-21fa-415c-a775-f56c3fca9112","Type":"ContainerStarted","Data":"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2"} Oct 31 00:27:28.046111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.046011 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" event={"ID":"50ff4ab8-21fa-415c-a775-f56c3fca9112","Type":"ContainerStarted","Data":"b871f8006af85395702500f1914fa171fb77cb540db98be4ceeb80bb360ede64"} Oct 31 00:27:28.046111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.046042 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:28.076336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.074769 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" podStartSLOduration=2.074751953 podStartE2EDuration="2.074751953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:28.069775471 +0000 UTC m=+1970.102871545" watchObservedRunningTime="2026-10-31 00:27:28.074751953 +0000 UTC m=+1970.107848025" Oct 31 00:27:28.193806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:28.193269 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" podStartSLOduration=2.193250437 podStartE2EDuration="2.193250437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:28.105657117 +0000 UTC m=+1970.138753188" watchObservedRunningTime="2026-10-31 00:27:28.193250437 +0000 UTC m=+1970.226346508" Oct 31 00:27:31.297690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.297654 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-11085-a749d3"] Oct 31 00:27:31.302741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.302713 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:31.308928 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.308897 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11085-a749d3"] Oct 31 00:27:31.318728 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.318701 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lr8lt\"" Oct 31 00:27:31.377327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.377290 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d9bcx\" (UniqueName: \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\") pod \"test-curl-11085-a749d3\" (UID: \"b3b08940-75ca-4735-8f65-2f3fdbe639b6\") " pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:31.479576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.479538 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d9bcx\" (UniqueName: \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\") pod \"test-curl-11085-a749d3\" (UID: \"b3b08940-75ca-4735-8f65-2f3fdbe639b6\") " pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:31.492745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.492715 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d9bcx\" (UniqueName: \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\") pod \"test-curl-11085-a749d3\" (UID: \"b3b08940-75ca-4735-8f65-2f3fdbe639b6\") " pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:31.618622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:31.618541 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:32.071706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:32.071677 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-11085-a749d3"] Oct 31 00:27:32.072286 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:32.072259 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb3b08940_75ca_4735_8f65_2f3fdbe639b6.slice/crio-2f1ab90595dbc192fa3e30ec8bdb4db805ce1cfc4e586c222fc4c0e360858de7 WatchSource:0}: Error finding container 2f1ab90595dbc192fa3e30ec8bdb4db805ce1cfc4e586c222fc4c0e360858de7: Status 404 returned error can't find the container with id 2f1ab90595dbc192fa3e30ec8bdb4db805ce1cfc4e586c222fc4c0e360858de7 Oct 31 00:27:32.098882 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:32.098845 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11085-a749d3" event={"ID":"b3b08940-75ca-4735-8f65-2f3fdbe639b6","Type":"ContainerStarted","Data":"2f1ab90595dbc192fa3e30ec8bdb4db805ce1cfc4e586c222fc4c0e360858de7"} Oct 31 00:27:33.109820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:33.109744 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-11085-a749d3" event={"ID":"b3b08940-75ca-4735-8f65-2f3fdbe639b6","Type":"ContainerStarted","Data":"08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07"} Oct 31 00:27:33.134667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:33.134618 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/test-curl-11085-a749d3" podStartSLOduration=2.13460139 podStartE2EDuration="2.13460139s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:33.133774045 +0000 UTC m=+1975.166870117" watchObservedRunningTime="2026-10-31 00:27:33.13460139 +0000 UTC m=+1975.167697460" Oct 31 00:27:33.188290 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:33.188252 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:27:34.021398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:34.020977 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-11085-a749d3"] Oct 31 00:27:34.029690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:34.029202 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-11085-a749d3"] Oct 31 00:27:35.010645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:35.010611 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:27:35.011140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:35.010847 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" containerID="cri-o://261d05543263a4e2fecebe04a46771ba41d2544e60b1e74a42779d912524366f" gracePeriod=30 Oct 31 00:27:35.237981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:35.237929 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/test-curl-11085-a749d3" podUID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" containerName="test-curl-11085-a749d3" containerID="cri-o://08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07" gracePeriod=2 Oct 31 00:27:37.808225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:37.808188 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:37.885415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:37.885326 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\" (UniqueName: \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\") pod \"b3b08940-75ca-4735-8f65-2f3fdbe639b6\" (UID: \"b3b08940-75ca-4735-8f65-2f3fdbe639b6\") " Oct 31 00:27:37.888445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:37.888411 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx" pod "b3b08940-75ca-4735-8f65-2f3fdbe639b6" (UID: "b3b08940-75ca-4735-8f65-2f3fdbe639b6"). InnerVolumeSpecName "kube-api-access-d9bcx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:37.988481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:37.988446 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d9bcx\" (UniqueName: \"kubernetes.io/projected/b3b08940-75ca-4735-8f65-2f3fdbe639b6-kube-api-access-d9bcx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:38.085199 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.085168 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" path="/var/lib/kubelet/pods/b3b08940-75ca-4735-8f65-2f3fdbe639b6/volumes" Oct 31 00:27:38.295683 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.295651 6234 generic.go:356] "Generic (PLEG): container finished" podID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" containerID="08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07" exitCode=137 Oct 31 00:27:38.295914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.295692 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-11085-a749d3" Oct 31 00:27:38.295914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.295751 6234 scope.go:122] "RemoveContainer" containerID="08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07" Oct 31 00:27:38.306752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.306729 6234 scope.go:122] "RemoveContainer" containerID="08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07" Oct 31 00:27:38.307136 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:38.307116 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07\": container with ID starting with 08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07 not found: ID does not exist" containerID="08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07" Oct 31 00:27:38.307214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:38.307145 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07"} err="failed to get container status \"08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07\": rpc error: code = NotFound desc = could not find container \"08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07\": container with ID starting with 08730fe77d9cd37b7112d456f5c6bdb6b6390dc336cef5655295b6b871ab5a07 not found: ID does not exist" Oct 31 00:27:39.057654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:39.057625 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:27:39.058130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:39.057702 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.426882 6234 generic.go:356] "Generic (PLEG): container finished" podID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerID="261d05543263a4e2fecebe04a46771ba41d2544e60b1e74a42779d912524366f" exitCode=0 Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.427089 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" event={"ID":"108ff184-1289-4ff7-9fb2-f7de9e5bf905","Type":"ContainerDied","Data":"261d05543263a4e2fecebe04a46771ba41d2544e60b1e74a42779d912524366f"} Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.427142 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" event={"ID":"108ff184-1289-4ff7-9fb2-f7de9e5bf905","Type":"ContainerDied","Data":"764f5becef65109eff22a50db07dfd86861f679fd7f0ce7e1af1510bd43d4c5a"} Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.427178 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="764f5becef65109eff22a50db07dfd86861f679fd7f0ce7e1af1510bd43d4c5a" Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.429166 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.430061 6234 generic.go:356] "Generic (PLEG): container finished" podID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerID="24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1" exitCode=0 Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.430157 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerDied","Data":"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1"} Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.430890 6234 scope.go:122] "RemoveContainer" containerID="24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1" Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.433485 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.433570 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:40.436848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.433618 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.548546 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.549610 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" containerName="test-curl-11085-a749d3" Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.549628 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" containerName="test-curl-11085-a749d3" Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.549639 6234 cpu_manager.go:414] "containerMap: removing container" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.549644 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.550114 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3b08940-75ca-4735-8f65-2f3fdbe639b6" containerName="test-curl-11085-a749d3" Oct 31 00:27:40.553079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.550133 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" containerName="istio-proxy" Oct 31 00:27:40.577471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.573994 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:27:40.577471 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.574124 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.581112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.579248 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-cc0fe1-serving-cert\"" Oct 31 00:27:40.587828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.587751 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.587935 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.587832 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588036 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588110 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588147 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588284 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588331 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588366 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.588496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.588471 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\") pod \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\" (UID: \"108ff184-1289-4ff7-9fb2-f7de9e5bf905\") " Oct 31 00:27:40.590394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.590366 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:40.590573 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.590552 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:40.591280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.591258 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:40.592058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.591803 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:40.592352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.592333 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:40.592735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.592716 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:40.594473 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.594446 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:40.595186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.595149 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:40.596163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.595872 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm" pod "108ff184-1289-4ff7-9fb2-f7de9e5bf905" (UID: "108ff184-1289-4ff7-9fb2-f7de9e5bf905"). InnerVolumeSpecName "kube-api-access-jrwvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:40.691432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691397 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.691432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691449 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.691711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691467 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.691711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691587 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f9vmq\" (UniqueName: \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691902 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691927 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691943 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691957 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691970 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691983 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.691996 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/108ff184-1289-4ff7-9fb2-f7de9e5bf905-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.692009 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jrwvm\" (UniqueName: \"kubernetes.io/projected/108ff184-1289-4ff7-9fb2-f7de9e5bf905-kube-api-access-jrwvm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.692053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.692023 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/108ff184-1289-4ff7-9fb2-f7de9e5bf905-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:40.793949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.793850 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.793949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.793908 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.793949 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.793929 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.794278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.793999 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f9vmq\" (UniqueName: \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.794565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.794540 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.796650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.796624 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.796762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.796696 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.807476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.807451 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f9vmq\" (UniqueName: \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\") pod \"maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:40.902162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:40.902125 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:41.208165 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:27:41.208134 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3aa20d4_7be0_48fb_9b28_c18b47f89b15.slice/crio-3cbff5597066b6ef3fb18ed7b5ee74cfc2dd0aecf1611f0059f3a8883eaeba57 WatchSource:0}: Error finding container 3cbff5597066b6ef3fb18ed7b5ee74cfc2dd0aecf1611f0059f3a8883eaeba57: Status 404 returned error can't find the container with id 3cbff5597066b6ef3fb18ed7b5ee74cfc2dd0aecf1611f0059f3a8883eaeba57 Oct 31 00:27:41.209265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.209242 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:27:41.464809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.464652 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" event={"ID":"d3aa20d4-7be0-48fb-9b28-c18b47f89b15","Type":"ContainerStarted","Data":"3cbff5597066b6ef3fb18ed7b5ee74cfc2dd0aecf1611f0059f3a8883eaeba57"} Oct 31 00:27:41.479808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.471819 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x" Oct 31 00:27:41.479808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.472050 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerStarted","Data":"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77"} Oct 31 00:27:41.527709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.527662 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:27:41.531995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.531969 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-p6w7x"] Oct 31 00:27:41.603284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.603253 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:27:41.604156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.604132 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:41.604272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:41.604179 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:42.098123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.098042 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="108ff184-1289-4ff7-9fb2-f7de9e5bf905" path="/var/lib/kubelet/pods/108ff184-1289-4ff7-9fb2-f7de9e5bf905/volumes" Oct 31 00:27:42.218811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.216893 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:27:42.483867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.483822 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" event={"ID":"d3aa20d4-7be0-48fb-9b28-c18b47f89b15","Type":"ContainerStarted","Data":"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1"} Oct 31 00:27:42.484336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.484140 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:42.509674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.509627 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" podStartSLOduration=9.044501724 podStartE2EDuration="9.509613754s" totalImagesPullingTime="465.11203ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:27:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:27:42.508995787 +0000 UTC m=+1984.542091893" watchObservedRunningTime="2026-10-31 00:27:42.509613754 +0000 UTC m=+1984.542709823" Oct 31 00:27:42.603745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.603713 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:42.603922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:42.603767 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:43.603464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:43.603430 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:43.603988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:43.603494 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:44.308880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:44.308843 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:27:44.608553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:44.608467 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:44.624957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:44.610681 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:45.603381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:45.603340 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:45.603572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:45.603400 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:46.603440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:46.603395 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:46.603881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:46.603465 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:47.604168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:47.604081 6234 patch_prober.go:29] interesting pod/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" start-of-body= Oct 31 00:27:47.604168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:47.604153 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.137:15021/healthz/ready\": dial tcp 10.134.0.137:15021: connect: connection refused" Oct 31 00:27:48.498430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:48.498401 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:27:48.608945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:48.608915 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:27:48.610328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:48.610307 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:27:48.775220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:48.775134 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:48.775403 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:48.775372 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" podUID="90907ec4-c51a-4a6b-b798-f7719ccfc821" containerName="maas-api" containerID="cri-o://6a1b78c0240e61bc5940add73e430963f97ad9fb760d7151cce6e0236b3ce3b0" gracePeriod=30 Oct 31 00:27:49.332022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.331896 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" podUID="b57f5056-547c-48ce-ac94-7748acca87cc" containerName="payload-pre-processing" containerID="cri-o://3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68" gracePeriod=25 Oct 31 00:27:49.684429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.674174 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:27:49.684429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.674494 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" containerID="cri-o://7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6" gracePeriod=30 Oct 31 00:27:49.691896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.691753 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:27:49.692427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.692377 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" containerID="cri-o://274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013" gracePeriod=30 Oct 31 00:27:49.703350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.702771 6234 generic.go:356] "Generic (PLEG): container finished" podID="90907ec4-c51a-4a6b-b798-f7719ccfc821" containerID="6a1b78c0240e61bc5940add73e430963f97ad9fb760d7151cce6e0236b3ce3b0" exitCode=0 Oct 31 00:27:49.703350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.703206 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" event={"ID":"90907ec4-c51a-4a6b-b798-f7719ccfc821","Type":"ContainerDied","Data":"6a1b78c0240e61bc5940add73e430963f97ad9fb760d7151cce6e0236b3ce3b0"} Oct 31 00:27:49.795620 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.795593 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:49.852396 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.852359 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\" (UniqueName: \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\") pod \"90907ec4-c51a-4a6b-b798-f7719ccfc821\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " Oct 31 00:27:49.852766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.852747 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\") pod \"90907ec4-c51a-4a6b-b798-f7719ccfc821\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " Oct 31 00:27:49.852924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.852877 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\") pod \"90907ec4-c51a-4a6b-b798-f7719ccfc821\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " Oct 31 00:27:49.852992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.852978 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\") pod \"90907ec4-c51a-4a6b-b798-f7719ccfc821\" (UID: \"90907ec4-c51a-4a6b-b798-f7719ccfc821\") " Oct 31 00:27:49.854352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.853966 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca" pod "90907ec4-c51a-4a6b-b798-f7719ccfc821" (UID: "90907ec4-c51a-4a6b-b798-f7719ccfc821"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:49.857257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.857227 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs" pod "90907ec4-c51a-4a6b-b798-f7719ccfc821" (UID: "90907ec4-c51a-4a6b-b798-f7719ccfc821"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:49.860276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.860249 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls" pod "90907ec4-c51a-4a6b-b798-f7719ccfc821" (UID: "90907ec4-c51a-4a6b-b798-f7719ccfc821"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:27:49.860458 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.860426 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf" pod "90907ec4-c51a-4a6b-b798-f7719ccfc821" (UID: "90907ec4-c51a-4a6b-b798-f7719ccfc821"). InnerVolumeSpecName "kube-api-access-h8rvf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:49.955979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.955936 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/90907ec4-c51a-4a6b-b798-f7719ccfc821-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:49.955979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.955968 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:49.955979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.955983 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/90907ec4-c51a-4a6b-b798-f7719ccfc821-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:49.956294 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:49.955993 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8rvf\" (UniqueName: \"kubernetes.io/projected/90907ec4-c51a-4a6b-b798-f7719ccfc821-kube-api-access-h8rvf\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:50.009614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.009575 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:27:50.058621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.057448 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\") pod \"b57f5056-547c-48ce-ac94-7748acca87cc\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " Oct 31 00:27:50.058621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.057713 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\" (UniqueName: \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\") pod \"b57f5056-547c-48ce-ac94-7748acca87cc\" (UID: \"b57f5056-547c-48ce-ac94-7748acca87cc\") " Oct 31 00:27:50.058621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.057987 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume" pod "b57f5056-547c-48ce-ac94-7748acca87cc" (UID: "b57f5056-547c-48ce-ac94-7748acca87cc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:50.058621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.058598 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b57f5056-547c-48ce-ac94-7748acca87cc-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:50.060599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.060570 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln" pod "b57f5056-547c-48ce-ac94-7748acca87cc" (UID: "b57f5056-547c-48ce-ac94-7748acca87cc"). InnerVolumeSpecName "kube-api-access-x7bln". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:50.169812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.164917 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x7bln\" (UniqueName: \"kubernetes.io/projected/b57f5056-547c-48ce-ac94-7748acca87cc-kube-api-access-x7bln\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:50.720741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.720713 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" Oct 31 00:27:50.721306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.720736 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp" event={"ID":"90907ec4-c51a-4a6b-b798-f7719ccfc821","Type":"ContainerDied","Data":"2bcb1bb3d01e7b1ecc3d1b5eaae121bdb9f2ca6e14851600abafee5823b9f9fa"} Oct 31 00:27:50.721306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.720853 6234 scope.go:122] "RemoveContainer" containerID="6a1b78c0240e61bc5940add73e430963f97ad9fb760d7151cce6e0236b3ce3b0" Oct 31 00:27:50.722404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.722381 6234 generic.go:356] "Generic (PLEG): container finished" podID="b57f5056-547c-48ce-ac94-7748acca87cc" containerID="3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68" exitCode=0 Oct 31 00:27:50.722516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.722430 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" Oct 31 00:27:50.722516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.722437 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" event={"ID":"b57f5056-547c-48ce-ac94-7748acca87cc","Type":"ContainerDied","Data":"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68"} Oct 31 00:27:50.722516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.722480 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr" event={"ID":"b57f5056-547c-48ce-ac94-7748acca87cc","Type":"ContainerDied","Data":"b5d402441499c4eefc4b279233b77eab16e3c37571b2d09394f0440116e6b568"} Oct 31 00:27:50.732648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.732627 6234 scope.go:122] "RemoveContainer" containerID="3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68" Oct 31 00:27:50.743582 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.743562 6234 scope.go:122] "RemoveContainer" containerID="3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68" Oct 31 00:27:50.743915 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:50.743887 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68\": container with ID starting with 3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68 not found: ID does not exist" containerID="3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68" Oct 31 00:27:50.744001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.743926 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68"} err="failed to get container status \"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68\": rpc error: code = NotFound desc = could not find container \"3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68\": container with ID starting with 3169ea2b766e3c8b70c846b1e1f5bf49ac124c197f3ccc3e835540dfc2724f68 not found: ID does not exist" Oct 31 00:27:50.755030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.755007 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:27:50.761064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.761042 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-08673d2c-5477bdf8dc-lsgpr"] Oct 31 00:27:50.774831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.774809 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:50.778979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:50.778957 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-08673d2c-65d858db4f-8jlvp"] Oct 31 00:27:52.086228 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:52.086192 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="90907ec4-c51a-4a6b-b798-f7719ccfc821" path="/var/lib/kubelet/pods/90907ec4-c51a-4a6b-b798-f7719ccfc821/volumes" Oct 31 00:27:52.086893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:52.086873 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b57f5056-547c-48ce-ac94-7748acca87cc" path="/var/lib/kubelet/pods/b57f5056-547c-48ce-ac94-7748acca87cc/volumes" Oct 31 00:27:55.019585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.019523 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:27:55.030663 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.030602 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.033983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.033956 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034028 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034095 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034164 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034336 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034325 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034357 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034393 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.034442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.034424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\") pod \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\" (UID: \"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653\") " Oct 31 00:27:55.035886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.035857 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.036070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.036051 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.036223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.036207 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.037077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.037041 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.037183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.037166 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:55.037977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.037949 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:55.038593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.038423 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.039142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.039120 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "kube-api-access-6fgvm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.039241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.039220 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token" pod "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" (UID: "bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.055875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.055852 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:27:55.136275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136239 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.136481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136311 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.136481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136349 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.136605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136515 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136849 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136899 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136938 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.136976 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.137029 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\") pod \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\" (UID: \"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0\") " Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.137047 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.137075 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.137501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.137289 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.138128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.137575 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:27:55.138299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138276 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138303 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138321 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138335 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6fgvm\" (UniqueName: \"kubernetes.io/projected/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-kube-api-access-6fgvm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138349 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138364 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138378 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138392 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138406 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138419 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138432 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138444 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.138708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138458 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.139133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.138743 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.140277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.140247 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:27:55.141156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.141129 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:27:55.141546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.141513 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.141648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.141549 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr" pod "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" (UID: "52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0"). InnerVolumeSpecName "kube-api-access-s5pcr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:27:55.240610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.240470 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5pcr\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-kube-api-access-s5pcr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.240610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.240505 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.240610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.240519 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.240610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.240527 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.240610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.240535 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:27:55.776630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.776591 6234 generic.go:356] "Generic (PLEG): container finished" podID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerID="274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013" exitCode=0 Oct 31 00:27:55.776941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.776697 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" Oct 31 00:27:55.776941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.776769 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" event={"ID":"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0","Type":"ContainerDied","Data":"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013"} Oct 31 00:27:55.777069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.776976 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c" event={"ID":"52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0","Type":"ContainerDied","Data":"2d4c40a85adebf5e047e663125b8ceb12f0441d8614856cb8ac4b620b687a202"} Oct 31 00:27:55.777069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.776998 6234 scope.go:122] "RemoveContainer" containerID="274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013" Oct 31 00:27:55.778665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.778644 6234 generic.go:356] "Generic (PLEG): container finished" podID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerID="7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6" exitCode=0 Oct 31 00:27:55.778776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.778699 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" Oct 31 00:27:55.778930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.778775 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" event={"ID":"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653","Type":"ContainerDied","Data":"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6"} Oct 31 00:27:55.779012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.778997 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b" event={"ID":"bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653","Type":"ContainerDied","Data":"a69d0540ce8580745ea11cf8f050de2a3dc8a643a3e4bb81805e1d6707a8f4f8"} Oct 31 00:27:55.790719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.790696 6234 scope.go:122] "RemoveContainer" containerID="274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013" Oct 31 00:27:55.791066 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:55.791042 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013\": container with ID starting with 274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013 not found: ID does not exist" containerID="274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013" Oct 31 00:27:55.791135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.791075 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013"} err="failed to get container status \"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013\": rpc error: code = NotFound desc = could not find container \"274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013\": container with ID starting with 274995cd0536b96545d88f5be386b5cb85ae0a89d1e250bffc6ce00b8a436013 not found: ID does not exist" Oct 31 00:27:55.791135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.791092 6234 scope.go:122] "RemoveContainer" containerID="7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6" Oct 31 00:27:55.803855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.803838 6234 scope.go:122] "RemoveContainer" containerID="7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6" Oct 31 00:27:55.804146 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:27:55.804126 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6\": container with ID starting with 7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6 not found: ID does not exist" containerID="7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6" Oct 31 00:27:55.804200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.804155 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6"} err="failed to get container status \"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6\": rpc error: code = NotFound desc = could not find container \"7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6\": container with ID starting with 7a22adfc92d65051808aa383916740cfdf8f5f11d9d33b7d9fb4e75d415f4bc6 not found: ID does not exist" Oct 31 00:27:55.814976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.814951 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:27:55.823723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.823696 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nw26c"] Oct 31 00:27:55.850219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.850187 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:27:55.854368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:55.854343 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-wbs6b"] Oct 31 00:27:56.085252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:56.085176 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" path="/var/lib/kubelet/pods/52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0/volumes" Oct 31 00:27:56.085818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:27:56.085771 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" path="/var/lib/kubelet/pods/bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653/volumes" Oct 31 00:28:03.437829 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:03.435679 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:28:08.094065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:08.093480 6234 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" secret="" err="secret \"e2e-api-a-719e12-openshift-default-dockercfg-vlkg9\" not found" Oct 31 00:28:08.107797 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:08.107748 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:28:08.233632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:08.233591 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:28:08.234168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:08.234129 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" containerID="cri-o://a442b5e166aa488498a5126ef5475d842f723317f41af0b63e27bee3f0b6c8ce" gracePeriod=30 Oct 31 00:28:08.681227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:08.681171 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" containerID="cri-o://a6fa73229fd3fb44928d460e264ee7921a1b91a5e115aa7766f2531d7daf2ae4" gracePeriod=30 Oct 31 00:28:11.340995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:11.340955 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:28:11.341549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:11.341517 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:11.424897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:11.424857 6234 patch_prober.go:29] interesting pod/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:28:11.425166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:11.425134 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:28:13.922140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:13.922097 6234 generic.go:356] "Generic (PLEG): container finished" podID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerID="a6fa73229fd3fb44928d460e264ee7921a1b91a5e115aa7766f2531d7daf2ae4" exitCode=0 Oct 31 00:28:13.922591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:13.922131 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" event={"ID":"62fc29d0-2c89-4029-baac-7c205319e5f0","Type":"ContainerDied","Data":"a6fa73229fd3fb44928d460e264ee7921a1b91a5e115aa7766f2531d7daf2ae4"} Oct 31 00:28:13.924341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:13.924309 6234 generic.go:356] "Generic (PLEG): container finished" podID="168583c7-7e54-45fa-8911-3627c8c96779" containerID="a442b5e166aa488498a5126ef5475d842f723317f41af0b63e27bee3f0b6c8ce" exitCode=0 Oct 31 00:28:13.924452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:13.924364 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" event={"ID":"168583c7-7e54-45fa-8911-3627c8c96779","Type":"ContainerDied","Data":"a442b5e166aa488498a5126ef5475d842f723317f41af0b63e27bee3f0b6c8ce"} Oct 31 00:28:14.273456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.273425 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:28:14.277990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.277961 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:28:14.325417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325377 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.325417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325472 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325497 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325542 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325582 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325622 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.325696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325667 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325743 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325853 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325902 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.325977 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.326266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326066 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.326266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326097 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.326266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326186 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.326266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326222 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326259 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.326521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326306 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.326521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326403 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\") pod \"62fc29d0-2c89-4029-baac-7c205319e5f0\" (UID: \"62fc29d0-2c89-4029-baac-7c205319e5f0\") " Oct 31 00:28:14.326521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326442 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\") pod \"168583c7-7e54-45fa-8911-3627c8c96779\" (UID: \"168583c7-7e54-45fa-8911-3627c8c96779\") " Oct 31 00:28:14.326521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326472 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.326731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326698 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.326939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.326917 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.327243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.327222 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:14.327452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.327430 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.328647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328214 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:14.328647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328247 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.328647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328354 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.328647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328507 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.328977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.327459 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/168583c7-7e54-45fa-8911-3627c8c96779-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.328977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328816 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.328977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328833 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.328977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328850 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.328977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.328866 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.330724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.330689 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "kube-api-access-b7hxx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.331388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.331343 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.332045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.332019 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:14.332197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.332178 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:14.332275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.332205 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.332333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.332294 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:14.334838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.334753 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn" pod "62fc29d0-2c89-4029-baac-7c205319e5f0" (UID: "62fc29d0-2c89-4029-baac-7c205319e5f0"). InnerVolumeSpecName "kube-api-access-hf6kn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.335012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.334988 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token" pod "168583c7-7e54-45fa-8911-3627c8c96779" (UID: "168583c7-7e54-45fa-8911-3627c8c96779"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430712 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf6kn\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-kube-api-access-hf6kn\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430760 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/62fc29d0-2c89-4029-baac-7c205319e5f0-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430794 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430810 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430823 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.430838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430836 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430849 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430862 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b7hxx\" (UniqueName: \"kubernetes.io/projected/168583c7-7e54-45fa-8911-3627c8c96779-kube-api-access-b7hxx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430876 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/168583c7-7e54-45fa-8911-3627c8c96779-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430889 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/168583c7-7e54-45fa-8911-3627c8c96779-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430904 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.431308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.430918 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/62fc29d0-2c89-4029-baac-7c205319e5f0-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:14.934854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.934824 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" Oct 31 00:28:14.935345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.934819 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn" event={"ID":"62fc29d0-2c89-4029-baac-7c205319e5f0","Type":"ContainerDied","Data":"0e02192592e98224eb36b07b912d4873d5fee745794d2cf372113c1096e97670"} Oct 31 00:28:14.935345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.935003 6234 scope.go:122] "RemoveContainer" containerID="a6fa73229fd3fb44928d460e264ee7921a1b91a5e115aa7766f2531d7daf2ae4" Oct 31 00:28:14.936630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.936613 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" Oct 31 00:28:14.936744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.936719 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq" event={"ID":"168583c7-7e54-45fa-8911-3627c8c96779","Type":"ContainerDied","Data":"a20f9a17ba5c623b7297c94f71492f97c0a2e5a96fa90fb0bb7fc410e2de11ce"} Oct 31 00:28:14.947922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.947874 6234 scope.go:122] "RemoveContainer" containerID="a442b5e166aa488498a5126ef5475d842f723317f41af0b63e27bee3f0b6c8ce" Oct 31 00:28:14.971366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.971330 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:28:14.982715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.982685 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-2cbwn"] Oct 31 00:28:14.994430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.994396 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:28:14.998989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:14.998961 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-719e12-openshift-default-59cc578b67-9c5vq"] Oct 31 00:28:16.086288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:16.086252 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="168583c7-7e54-45fa-8911-3627c8c96779" path="/var/lib/kubelet/pods/168583c7-7e54-45fa-8911-3627c8c96779/volumes" Oct 31 00:28:16.086862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:16.086847 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" path="/var/lib/kubelet/pods/62fc29d0-2c89-4029-baac-7c205319e5f0/volumes" Oct 31 00:28:18.960756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.960719 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:28:18.994619 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.994582 6234 generic.go:356] "Generic (PLEG): container finished" podID="92427017-66b5-49af-8c7f-3c8867b057fe" containerID="2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624" exitCode=0 Oct 31 00:28:18.994925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.994904 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerDied","Data":"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624"} Oct 31 00:28:18.996074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.996043 6234 scope.go:122] "RemoveContainer" containerID="2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624" Oct 31 00:28:18.998977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.998559 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:28:18.998977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.998661 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:28:18.998977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:18.998708 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:28:20.007030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:20.006987 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerStarted","Data":"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084"} Oct 31 00:28:20.309151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:20.309068 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:28:20.310599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:20.310573 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:20.310711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:20.310619 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:21.309262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:21.309223 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:21.309876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:21.309279 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:22.309413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:22.309373 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:22.309936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:22.309434 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:23.310145 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:23.310105 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:23.310565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:23.310171 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:23.963813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:23.962847 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" podUID="9370837c-e0ba-4889-8d68-136421253993" containerName="payload-pre-processing" containerID="cri-o://2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1" gracePeriod=25 Oct 31 00:28:24.263887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.251899 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:28:24.309407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.309374 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:24.309578 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.309429 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:24.326071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.326045 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:28:24.400720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.400678 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\" (UniqueName: \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\") pod \"9370837c-e0ba-4889-8d68-136421253993\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " Oct 31 00:28:24.400941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.400763 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\") pod \"9370837c-e0ba-4889-8d68-136421253993\" (UID: \"9370837c-e0ba-4889-8d68-136421253993\") " Oct 31 00:28:24.401207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.401178 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume" pod "9370837c-e0ba-4889-8d68-136421253993" (UID: "9370837c-e0ba-4889-8d68-136421253993"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:24.401502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.401482 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9370837c-e0ba-4889-8d68-136421253993-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:24.403115 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.403091 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr" pod "9370837c-e0ba-4889-8d68-136421253993" (UID: "9370837c-e0ba-4889-8d68-136421253993"). InnerVolumeSpecName "kube-api-access-cvttr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:24.503653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:24.503472 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cvttr\" (UniqueName: \"kubernetes.io/projected/9370837c-e0ba-4889-8d68-136421253993-kube-api-access-cvttr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:25.126140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.126103 6234 generic.go:356] "Generic (PLEG): container finished" podID="9370837c-e0ba-4889-8d68-136421253993" containerID="2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1" exitCode=0 Oct 31 00:28:25.126558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.126531 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" event={"ID":"9370837c-e0ba-4889-8d68-136421253993","Type":"ContainerDied","Data":"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1"} Oct 31 00:28:25.128699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.126678 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" Oct 31 00:28:25.128699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.126723 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb" event={"ID":"9370837c-e0ba-4889-8d68-136421253993","Type":"ContainerDied","Data":"68ec0747154b060dc143456ffd1449147f1864b999d0cc4a3b42c1834fc16128"} Oct 31 00:28:25.128699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.126764 6234 scope.go:122] "RemoveContainer" containerID="2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1" Oct 31 00:28:25.167426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.166713 6234 scope.go:122] "RemoveContainer" containerID="2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1" Oct 31 00:28:25.167426 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:25.167324 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1\": container with ID starting with 2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1 not found: ID does not exist" containerID="2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1" Oct 31 00:28:25.167426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.167360 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1"} err="failed to get container status \"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1\": rpc error: code = NotFound desc = could not find container \"2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1\": container with ID starting with 2e265f61916b10fd988415b0bea1229315de4079b2d09f22b9b1bdf9c88569a1 not found: ID does not exist" Oct 31 00:28:25.199158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.199127 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:28:25.203150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.203129 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-7b1fc7-6f5fcbb6f9-gddbb"] Oct 31 00:28:25.309720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.309688 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" start-of-body= Oct 31 00:28:25.309941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:25.309747 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.144:15021/healthz/ready\": dial tcp 10.134.0.144:15021: connect: connection refused" Oct 31 00:28:26.083719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:26.083686 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9370837c-e0ba-4889-8d68-136421253993" path="/var/lib/kubelet/pods/9370837c-e0ba-4889-8d68-136421253993/volumes" Oct 31 00:28:26.388153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:26.388066 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:28:26.391005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:26.390979 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:28:27.090433 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:27.090403 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:28:27.100807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:27.100128 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:28:32.090876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.090842 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" podUID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" containerName="payload-processing" containerID="cri-o://3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b" gracePeriod=25 Oct 31 00:28:32.100570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.100537 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerName="payload-pre-processing" containerID="cri-o://b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0" gracePeriod=25 Oct 31 00:28:32.446564 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:28:32.317426 6234 logging.go:55] [core] [Channel #3914 SubChannel #3915]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.138:50052", ServerName: "10.134.0.138:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.138:50052: connect: connection refused" Oct 31 00:28:32.698005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.697935 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:28:32.748931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.748904 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\") pod \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " Oct 31 00:28:32.749218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.749196 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume" pod "fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" (UID: "fc92ce56-f849-4ea0-8aaf-b28ed7e244b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:32.749651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.749627 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.802365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.802332 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:28:32.863812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.855404 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\" (UniqueName: \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\") pod \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\" (UID: \"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3\") " Oct 31 00:28:32.921389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.921356 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs" pod "fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" (UID: "fc92ce56-f849-4ea0-8aaf-b28ed7e244b3"). InnerVolumeSpecName "kube-api-access-tdfcs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:32.966624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.966541 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\") pod \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " Oct 31 00:28:32.966827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.966727 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\" (UniqueName: \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\") pod \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\" (UID: \"aa8b4f3c-701d-4d1c-aa78-827fa3d28060\") " Oct 31 00:28:32.966978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.966953 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume" pod "aa8b4f3c-701d-4d1c-aa78-827fa3d28060" (UID: "aa8b4f3c-701d-4d1c-aa78-827fa3d28060"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:32.967501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.967477 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tdfcs\" (UniqueName: \"kubernetes.io/projected/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3-kube-api-access-tdfcs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.967584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.967506 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:32.968964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:32.968945 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v" pod "aa8b4f3c-701d-4d1c-aa78-827fa3d28060" (UID: "aa8b4f3c-701d-4d1c-aa78-827fa3d28060"). InnerVolumeSpecName "kube-api-access-ppd9v". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:33.071176 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.071135 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppd9v\" (UniqueName: \"kubernetes.io/projected/aa8b4f3c-701d-4d1c-aa78-827fa3d28060-kube-api-access-ppd9v\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:33.163415 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:33.163376 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:28:33.173494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.173460 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:28:33.210878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.210844 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:28:33.212140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.212113 6234 generic.go:356] "Generic (PLEG): container finished" podID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerID="b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0" exitCode=0 Oct 31 00:28:33.212301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.212181 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" Oct 31 00:28:33.212301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.212195 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" event={"ID":"aa8b4f3c-701d-4d1c-aa78-827fa3d28060","Type":"ContainerDied","Data":"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0"} Oct 31 00:28:33.212301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.212245 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" event={"ID":"aa8b4f3c-701d-4d1c-aa78-827fa3d28060","Type":"ContainerDied","Data":"8f337e4b7e79e3cf7bbfd9dd5cc138c9d4b35d4ff8ac072f61872ac33ec0c741"} Oct 31 00:28:33.212301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.212273 6234 scope.go:122] "RemoveContainer" containerID="b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0" Oct 31 00:28:33.213747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.213729 6234 generic.go:356] "Generic (PLEG): container finished" podID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" containerID="3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b" exitCode=0 Oct 31 00:28:33.213921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.213766 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" event={"ID":"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3","Type":"ContainerDied","Data":"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b"} Oct 31 00:28:33.213921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.213824 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" Oct 31 00:28:33.213921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.213842 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c" event={"ID":"fc92ce56-f849-4ea0-8aaf-b28ed7e244b3","Type":"ContainerDied","Data":"a450a204b577abf3614b6dd19c6fb3bb115c9792cecdaccd1190da0dbdc1a4b3"} Oct 31 00:28:33.225810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.225774 6234 scope.go:122] "RemoveContainer" containerID="b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0" Oct 31 00:28:33.226123 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:33.226106 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0\": container with ID starting with b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0 not found: ID does not exist" containerID="b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0" Oct 31 00:28:33.226172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.226132 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0"} err="failed to get container status \"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0\": rpc error: code = NotFound desc = could not find container \"b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0\": container with ID starting with b5ac963653a00a7654d830850e8644e9b990bae86a3c6ac7cdf4b2867643fca0 not found: ID does not exist" Oct 31 00:28:33.226172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.226148 6234 scope.go:122] "RemoveContainer" containerID="3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b" Oct 31 00:28:33.238347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.238305 6234 scope.go:122] "RemoveContainer" containerID="3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b" Oct 31 00:28:33.238728 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:33.238696 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b\": container with ID starting with 3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b not found: ID does not exist" containerID="3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b" Oct 31 00:28:33.238850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.238737 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b"} err="failed to get container status \"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b\": rpc error: code = NotFound desc = could not find container \"3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b\": container with ID starting with 3ad85941f9ab85e322096b03252afaf6570bed5ee1cc7b8a3fa77784a393897b not found: ID does not exist" Oct 31 00:28:33.292497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.292460 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:28:33.296709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.296684 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-d609e2-5b9d4bfff-j4n4c"] Oct 31 00:28:33.317060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.317021 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.138:50052\" within 1s: context deadline exceeded" Oct 31 00:28:33.347427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.347396 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:28:33.369084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:33.368977 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-d609e2-b464d7dff-fh8jk"] Oct 31 00:28:34.100466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:34.100429 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" path="/var/lib/kubelet/pods/aa8b4f3c-701d-4d1c-aa78-827fa3d28060/volumes" Oct 31 00:28:34.101476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:34.101447 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" path="/var/lib/kubelet/pods/fc92ce56-f849-4ea0-8aaf-b28ed7e244b3/volumes" Oct 31 00:28:38.174249 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.174184 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" podUID="08486c3c-312d-4813-af56-8b8a597b2a7a" containerName="payload-processing" containerID="cri-o://60216f290ff2765157db08166a33f2f35b01a413619e8f5095a35a8f05df0b2a" gracePeriod=25 Oct 31 00:28:38.211177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.211118 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" podUID="e876ba6b-d1fe-497c-8714-bda90849eefd" containerName="payload-pre-processing" containerID="cri-o://dc6cf0bd14a072c80a6eef8581211289a2485b05a02d5fa2495873ac676b9b42" gracePeriod=25 Oct 31 00:28:38.370835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.370674 6234 generic.go:356] "Generic (PLEG): container finished" podID="08486c3c-312d-4813-af56-8b8a597b2a7a" containerID="60216f290ff2765157db08166a33f2f35b01a413619e8f5095a35a8f05df0b2a" exitCode=0 Oct 31 00:28:38.370835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.370684 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" event={"ID":"08486c3c-312d-4813-af56-8b8a597b2a7a","Type":"ContainerDied","Data":"60216f290ff2765157db08166a33f2f35b01a413619e8f5095a35a8f05df0b2a"} Oct 31 00:28:38.372681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.372658 6234 generic.go:356] "Generic (PLEG): container finished" podID="e876ba6b-d1fe-497c-8714-bda90849eefd" containerID="dc6cf0bd14a072c80a6eef8581211289a2485b05a02d5fa2495873ac676b9b42" exitCode=0 Oct 31 00:28:38.372854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.372691 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" event={"ID":"e876ba6b-d1fe-497c-8714-bda90849eefd","Type":"ContainerDied","Data":"dc6cf0bd14a072c80a6eef8581211289a2485b05a02d5fa2495873ac676b9b42"} Oct 31 00:28:38.546395 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.546367 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:28:38.586565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.586541 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:28:38.614837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.614806 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\" (UniqueName: \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\") pod \"08486c3c-312d-4813-af56-8b8a597b2a7a\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " Oct 31 00:28:38.614978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.614938 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\" (UniqueName: \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\") pod \"e876ba6b-d1fe-497c-8714-bda90849eefd\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " Oct 31 00:28:38.615156 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.615137 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\") pod \"08486c3c-312d-4813-af56-8b8a597b2a7a\" (UID: \"08486c3c-312d-4813-af56-8b8a597b2a7a\") " Oct 31 00:28:38.615250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.615232 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\") pod \"e876ba6b-d1fe-497c-8714-bda90849eefd\" (UID: \"e876ba6b-d1fe-497c-8714-bda90849eefd\") " Oct 31 00:28:38.615526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.615499 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume" pod "08486c3c-312d-4813-af56-8b8a597b2a7a" (UID: "08486c3c-312d-4813-af56-8b8a597b2a7a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:38.615693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.615646 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume" pod "e876ba6b-d1fe-497c-8714-bda90849eefd" (UID: "e876ba6b-d1fe-497c-8714-bda90849eefd"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:38.616068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.616020 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/08486c3c-312d-4813-af56-8b8a597b2a7a-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:38.616068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.616049 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e876ba6b-d1fe-497c-8714-bda90849eefd-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:38.617446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.617417 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw" pod "08486c3c-312d-4813-af56-8b8a597b2a7a" (UID: "08486c3c-312d-4813-af56-8b8a597b2a7a"). InnerVolumeSpecName "kube-api-access-hhhxw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:38.617920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.617899 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c" pod "e876ba6b-d1fe-497c-8714-bda90849eefd" (UID: "e876ba6b-d1fe-497c-8714-bda90849eefd"). InnerVolumeSpecName "kube-api-access-std5c". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:38.717925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.717840 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-std5c\" (UniqueName: \"kubernetes.io/projected/e876ba6b-d1fe-497c-8714-bda90849eefd-kube-api-access-std5c\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:38.717925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:38.717883 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hhhxw\" (UniqueName: \"kubernetes.io/projected/08486c3c-312d-4813-af56-8b8a597b2a7a-kube-api-access-hhhxw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:39.382848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.382820 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" Oct 31 00:28:39.383289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.382819 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc" event={"ID":"e876ba6b-d1fe-497c-8714-bda90849eefd","Type":"ContainerDied","Data":"aa920c2e5510a147ef416ba179360bebe573a802ff72c1e842f6931c26af258d"} Oct 31 00:28:39.383289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.382960 6234 scope.go:122] "RemoveContainer" containerID="dc6cf0bd14a072c80a6eef8581211289a2485b05a02d5fa2495873ac676b9b42" Oct 31 00:28:39.384644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.384625 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" Oct 31 00:28:39.384917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.384664 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4" event={"ID":"08486c3c-312d-4813-af56-8b8a597b2a7a","Type":"ContainerDied","Data":"82882309f1de1b645cb1ecc10fd04708cef87f11aa446863c9ef393d2a9f1ae7"} Oct 31 00:28:39.395761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.395733 6234 scope.go:122] "RemoveContainer" containerID="60216f290ff2765157db08166a33f2f35b01a413619e8f5095a35a8f05df0b2a" Oct 31 00:28:39.420545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.420520 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:28:39.430618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.430592 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w1-0df3f8-584678d5fc-pw9h4"] Oct 31 00:28:39.446198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.446170 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:28:39.450132 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:39.450109 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w1-0df3f8-5886cd6f75-rjvxc"] Oct 31 00:28:40.084561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:40.084524 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="08486c3c-312d-4813-af56-8b8a597b2a7a" path="/var/lib/kubelet/pods/08486c3c-312d-4813-af56-8b8a597b2a7a/volumes" Oct 31 00:28:40.085059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:40.085045 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e876ba6b-d1fe-497c-8714-bda90849eefd" path="/var/lib/kubelet/pods/e876ba6b-d1fe-497c-8714-bda90849eefd/volumes" Oct 31 00:28:40.669323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:40.669286 6234 scope.go:122] "RemoveContainer" containerID="261d05543263a4e2fecebe04a46771ba41d2544e60b1e74a42779d912524366f" Oct 31 00:28:45.324588 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:45.324535 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:28:45.325121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:45.324901 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" containerID="cri-o://a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a" gracePeriod=30 Oct 31 00:28:46.226914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.226875 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:28:46.228219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228187 6234 cpu_manager.go:414] "containerMap: removing container" podUID="08486c3c-312d-4813-af56-8b8a597b2a7a" containerName="payload-processing" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228225 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="08486c3c-312d-4813-af56-8b8a597b2a7a" containerName="payload-processing" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228248 6234 cpu_manager.go:414] "containerMap: removing container" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228256 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228277 6234 cpu_manager.go:414] "containerMap: removing container" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerName="payload-pre-processing" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228287 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerName="payload-pre-processing" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228304 6234 cpu_manager.go:414] "containerMap: removing container" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228311 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228343 6234 cpu_manager.go:414] "containerMap: removing container" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" Oct 31 00:28:46.228351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228351 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228364 6234 cpu_manager.go:414] "containerMap: removing container" podUID="90907ec4-c51a-4a6b-b798-f7719ccfc821" containerName="maas-api" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228370 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="90907ec4-c51a-4a6b-b798-f7719ccfc821" containerName="maas-api" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228393 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b57f5056-547c-48ce-ac94-7748acca87cc" containerName="payload-pre-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228400 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b57f5056-547c-48ce-ac94-7748acca87cc" containerName="payload-pre-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228425 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" containerName="payload-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228432 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" containerName="payload-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228452 6234 cpu_manager.go:414] "containerMap: removing container" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228459 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228489 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9370837c-e0ba-4889-8d68-136421253993" containerName="payload-pre-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228497 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9370837c-e0ba-4889-8d68-136421253993" containerName="payload-pre-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228510 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e876ba6b-d1fe-497c-8714-bda90849eefd" containerName="payload-pre-processing" Oct 31 00:28:46.228646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228519 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e876ba6b-d1fe-497c-8714-bda90849eefd" containerName="payload-pre-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228706 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc92ce56-f849-4ea0-8aaf-b28ed7e244b3" containerName="payload-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228721 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb5a7af4-a5e7-444a-a4f6-bc2d60bb8653" containerName="istio-proxy" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228735 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9370837c-e0ba-4889-8d68-136421253993" containerName="payload-pre-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228745 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e876ba6b-d1fe-497c-8714-bda90849eefd" containerName="payload-pre-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228758 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="08486c3c-312d-4813-af56-8b8a597b2a7a" containerName="payload-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228768 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="52c9f3ff-aaab-4e1b-9b8a-ff6c65fa37e0" containerName="istio-proxy" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228775 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="aa8b4f3c-701d-4d1c-aa78-827fa3d28060" containerName="payload-pre-processing" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228808 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="168583c7-7e54-45fa-8911-3627c8c96779" containerName="istio-proxy" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228822 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="62fc29d0-2c89-4029-baac-7c205319e5f0" containerName="istio-proxy" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228832 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="90907ec4-c51a-4a6b-b798-f7719ccfc821" containerName="maas-api" Oct 31 00:28:46.229076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.228844 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b57f5056-547c-48ce-ac94-7748acca87cc" containerName="payload-pre-processing" Oct 31 00:28:46.235472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.235451 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:28:46.235597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.235578 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.241265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.241244 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.277115 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.277059 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:28:46.281725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.281696 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:28:46.284717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.284694 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-31108d\"" Oct 31 00:28:46.294879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.294848 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-31108d-dockercfg-5pb7z\"" Oct 31 00:28:46.629997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.629908 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.630766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.630727 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j66nt\" (UniqueName: \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.732038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.732001 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g6fm5\" (UniqueName: \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.732242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.732051 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.732242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.732180 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j66nt\" (UniqueName: \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.732337 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.732323 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.732860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.732844 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.750602 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.750574 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j66nt\" (UniqueName: \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\") pod \"payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:46.834187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.834153 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.834384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.834368 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g6fm5\" (UniqueName: \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.834970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.834952 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.850994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.850962 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g6fm5\" (UniqueName: \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\") pod \"payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.853797 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.853752 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:46.860144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:46.860123 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:47.404841 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:28:47.404809 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod639d6597_1d61_4757_8dfe_2e4ddf1e57f3.slice/crio-c52a6c5b9d8afa21c9c03fde5ae1e73c9b1eb41178939d33253d8b8ea1237f9e WatchSource:0}: Error finding container c52a6c5b9d8afa21c9c03fde5ae1e73c9b1eb41178939d33253d8b8ea1237f9e: Status 404 returned error can't find the container with id c52a6c5b9d8afa21c9c03fde5ae1e73c9b1eb41178939d33253d8b8ea1237f9e Oct 31 00:28:47.405500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.405475 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:28:47.420511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.420485 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:28:47.421526 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:28:47.421502 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf8eda208_4692_45bd_ba91_9eb470e5104c.slice/crio-b1efb3e989c73d387e9d39896ede2048d098ccec0c7de13e2fa4056b8af21654 WatchSource:0}: Error finding container b1efb3e989c73d387e9d39896ede2048d098ccec0c7de13e2fa4056b8af21654: Status 404 returned error can't find the container with id b1efb3e989c73d387e9d39896ede2048d098ccec0c7de13e2fa4056b8af21654 Oct 31 00:28:47.770357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.766869 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" event={"ID":"639d6597-1d61-4757-8dfe-2e4ddf1e57f3","Type":"ContainerStarted","Data":"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d"} Oct 31 00:28:47.770357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.766942 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" event={"ID":"639d6597-1d61-4757-8dfe-2e4ddf1e57f3","Type":"ContainerStarted","Data":"c52a6c5b9d8afa21c9c03fde5ae1e73c9b1eb41178939d33253d8b8ea1237f9e"} Oct 31 00:28:47.770357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.768332 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:47.805895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.800359 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" event={"ID":"f8eda208-4692-45bd-ba91-9eb470e5104c","Type":"ContainerStarted","Data":"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487"} Oct 31 00:28:47.805895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.800425 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" event={"ID":"f8eda208-4692-45bd-ba91-9eb470e5104c","Type":"ContainerStarted","Data":"b1efb3e989c73d387e9d39896ede2048d098ccec0c7de13e2fa4056b8af21654"} Oct 31 00:28:47.805895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.801608 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:28:47.948862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:47.945890 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" podStartSLOduration=1.9458669629999998 podStartE2EDuration="1.945866963s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.918207158 +0000 UTC m=+2049.951303228" watchObservedRunningTime="2026-10-31 00:28:47.945866963 +0000 UTC m=+2049.978963035" Oct 31 00:28:48.053870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:48.042496 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" podStartSLOduration=2.042480024 podStartE2EDuration="2.042480024s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:28:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:28:47.837352235 +0000 UTC m=+2049.870448306" watchObservedRunningTime="2026-10-31 00:28:48.042480024 +0000 UTC m=+2050.075576096" Oct 31 00:28:49.890905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:49.890872 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:28:49.891331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:49.891075 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" containerID="cri-o://d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7" gracePeriod=30 Oct 31 00:28:50.617466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.617442 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:28:50.675518 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675488 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675539 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675588 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675677 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675753 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675798 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.675894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675819 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.676053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.675951 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.676053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.676027 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\") pod \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\" (UID: \"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88\") " Oct 31 00:28:50.676164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.676137 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:50.676509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.676464 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:50.676827 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.676796 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:50.677012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.676988 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.677087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.677019 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.677087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.677034 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.677240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.677221 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:50.677415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.677389 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:50.678971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.678942 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:50.683822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.679720 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:50.683822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.679749 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:50.683822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.679876 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7" pod "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" (UID: "db0d1e8b-f5dc-48a1-889e-f15d91e9eb88"). InnerVolumeSpecName "kube-api-access-4qrs7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:50.778561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778529 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.778561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778558 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.778561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778568 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4qrs7\" (UniqueName: \"kubernetes.io/projected/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-kube-api-access-4qrs7\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.778806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778579 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.778806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778588 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.778806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.778597 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:50.858081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.857978 6234 generic.go:356] "Generic (PLEG): container finished" podID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerID="a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a" exitCode=0 Oct 31 00:28:50.858081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.858057 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" Oct 31 00:28:50.858327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.858082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerDied","Data":"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a"} Oct 31 00:28:50.858327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.858151 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f" event={"ID":"db0d1e8b-f5dc-48a1-889e-f15d91e9eb88","Type":"ContainerDied","Data":"d4bb8642575343739424e4ff8c4db7fff44e598a51c8058995dbbeeda23bdcdd"} Oct 31 00:28:50.858327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.858185 6234 scope.go:122] "RemoveContainer" containerID="a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a" Oct 31 00:28:50.870366 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.870348 6234 scope.go:122] "RemoveContainer" containerID="ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363" Oct 31 00:28:50.880146 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.880127 6234 scope.go:122] "RemoveContainer" containerID="a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a" Oct 31 00:28:50.880389 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:50.880370 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a\": container with ID starting with a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a not found: ID does not exist" containerID="a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a" Oct 31 00:28:50.880445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.880397 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a"} err="failed to get container status \"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a\": rpc error: code = NotFound desc = could not find container \"a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a\": container with ID starting with a7084238c05794aecb1b60c0785c7303a6ce589e9c2b8aae908f10b20423456a not found: ID does not exist" Oct 31 00:28:50.880445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.880413 6234 scope.go:122] "RemoveContainer" containerID="ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363" Oct 31 00:28:50.880654 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:50.880631 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363\": container with ID starting with ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363 not found: ID does not exist" containerID="ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363" Oct 31 00:28:50.880712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.880664 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363"} err="failed to get container status \"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363\": rpc error: code = NotFound desc = could not find container \"ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363\": container with ID starting with ad7d6f4450e121c44ee7c54a02fb1cf8accb9eeb3e02516c14ed6d26a502a363 not found: ID does not exist" Oct 31 00:28:50.902594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.902560 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:28:50.906707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:50.906684 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-08673d2c-openshift-default-bcf78c9b9-tl99f"] Oct 31 00:28:52.083839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:52.083764 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" path="/var/lib/kubelet/pods/db0d1e8b-f5dc-48a1-889e-f15d91e9eb88/volumes" Oct 31 00:28:55.540286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.540245 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:28:55.729878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.729844 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.729896 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730065 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730131 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730136 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:55.730224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730151 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:55.730386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730237 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730271 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730307 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730386 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730376 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:55.730618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730385 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\") pod \"4abcb2ca-7779-4387-b0af-1579b448227b\" (UID: \"4abcb2ca-7779-4387-b0af-1579b448227b\") " Oct 31 00:28:55.730732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730661 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:55.730732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.730688 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:28:55.731579 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.731549 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.731682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.731583 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.731682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.731599 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.731682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.731649 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4abcb2ca-7779-4387-b0af-1579b448227b-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.731682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.731665 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.732769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.732747 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "kube-api-access-crzkd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:55.732880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.732820 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:28:55.733040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.732985 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:28:55.733040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.733029 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo" pod "4abcb2ca-7779-4387-b0af-1579b448227b" (UID: "4abcb2ca-7779-4387-b0af-1579b448227b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:28:55.833577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.833545 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4abcb2ca-7779-4387-b0af-1579b448227b-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.833577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.833573 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.833577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.833582 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crzkd\" (UniqueName: \"kubernetes.io/projected/4abcb2ca-7779-4387-b0af-1579b448227b-kube-api-access-crzkd\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.833832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.833592 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4abcb2ca-7779-4387-b0af-1579b448227b-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:28:55.978865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.978832 6234 generic.go:356] "Generic (PLEG): container finished" podID="4abcb2ca-7779-4387-b0af-1579b448227b" containerID="d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7" exitCode=0 Oct 31 00:28:55.979043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.978891 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" event={"ID":"4abcb2ca-7779-4387-b0af-1579b448227b","Type":"ContainerDied","Data":"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7"} Oct 31 00:28:55.979043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.978952 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" event={"ID":"4abcb2ca-7779-4387-b0af-1579b448227b","Type":"ContainerDied","Data":"f56f6a371370738f410174a1357a306ab6f15d7efa3aedc64a803f3cec467b53"} Oct 31 00:28:55.979043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.978892 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k" Oct 31 00:28:55.979043 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.978971 6234 scope.go:122] "RemoveContainer" containerID="d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7" Oct 31 00:28:55.990665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.990647 6234 scope.go:122] "RemoveContainer" containerID="d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7" Oct 31 00:28:55.991059 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:28:55.991037 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7\": container with ID starting with d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7 not found: ID does not exist" containerID="d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7" Oct 31 00:28:55.991154 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:55.991072 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7"} err="failed to get container status \"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7\": rpc error: code = NotFound desc = could not find container \"d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7\": container with ID starting with d6e3c04e88bf92ff306a1e023a15dea184700b6f39640d949962beeb874851f7 not found: ID does not exist" Oct 31 00:28:56.032002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:56.031970 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:28:56.041622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:56.041591 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-w744k"] Oct 31 00:28:56.085030 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:56.084998 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" path="/var/lib/kubelet/pods/4abcb2ca-7779-4387-b0af-1579b448227b/volumes" Oct 31 00:28:57.830673 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.830626 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:28:57.831157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.830954 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" containerID="cri-o://41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176" gracePeriod=30 Oct 31 00:28:57.836338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.836311 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:28:57.836567 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.836536 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" containerID="cri-o://ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7" gracePeriod=30 Oct 31 00:28:57.840624 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.840604 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:28:57.840798 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:57.840765 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" containerID="cri-o://7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b" gracePeriod=30 Oct 31 00:28:59.849355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:59.849324 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:28:59.849939 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:28:59.849400 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:29:02.566123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:02.566087 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:02.566533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:02.566144 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:02.942287 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:02.942250 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbea6a00f_672b_4a94_971b_106c506294fd.slice/crio-7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6fa69024_0355_4478_afa2_ef91dffc4adf.slice/crio-conmon-ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7.scope\": RecentStats: unable to find data in memory cache]" Oct 31 00:29:03.175277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.175246 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:29:03.183250 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:03.183196 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:29:03.196852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.196815 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:29:03.200342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.199733 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerDied","Data":"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176"} Oct 31 00:29:03.200342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.199765 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" Oct 31 00:29:03.200342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.199945 6234 scope.go:122] "RemoveContainer" containerID="41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176" Oct 31 00:29:03.200885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.199181 6234 generic.go:356] "Generic (PLEG): container finished" podID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerID="41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176" exitCode=0 Oct 31 00:29:03.200885 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.200828 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp" event={"ID":"277e0d55-a194-44cf-a75f-771a24a9fb65","Type":"ContainerDied","Data":"88ba7b119c9a8790a5aed1d0895f0021a6f3579b79dcab014d7bff34eb31366e"} Oct 31 00:29:03.211611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.211581 6234 generic.go:356] "Generic (PLEG): container finished" podID="bea6a00f-672b-4a94-971b-106c506294fd" containerID="7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b" exitCode=0 Oct 31 00:29:03.211756 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.211668 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" event={"ID":"bea6a00f-672b-4a94-971b-106c506294fd","Type":"ContainerDied","Data":"7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b"} Oct 31 00:29:03.213761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.213738 6234 generic.go:356] "Generic (PLEG): container finished" podID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerID="ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7" exitCode=0 Oct 31 00:29:03.213916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.213774 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" event={"ID":"6fa69024-0355-4478-afa2-ef91dffc4adf","Type":"ContainerDied","Data":"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7"} Oct 31 00:29:03.213916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.213870 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" event={"ID":"6fa69024-0355-4478-afa2-ef91dffc4adf","Type":"ContainerDied","Data":"8125abdd85d176c5c2f079e124be2515f30727425ef832a08a28b65c5ca554a8"} Oct 31 00:29:03.214040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.213947 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh" Oct 31 00:29:03.222498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.222472 6234 scope.go:122] "RemoveContainer" containerID="48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9" Oct 31 00:29:03.237515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.237461 6234 scope.go:122] "RemoveContainer" containerID="41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176" Oct 31 00:29:03.237980 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:03.237950 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176\": container with ID starting with 41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176 not found: ID does not exist" containerID="41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176" Oct 31 00:29:03.238102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.237994 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176"} err="failed to get container status \"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176\": rpc error: code = NotFound desc = could not find container \"41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176\": container with ID starting with 41fbcd9fd046b6ffd90cd280abdbee5f41b28d3da801fcae7e4875b773269176 not found: ID does not exist" Oct 31 00:29:03.238102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.238020 6234 scope.go:122] "RemoveContainer" containerID="48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9" Oct 31 00:29:03.239205 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:03.238362 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9\": container with ID starting with 48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9 not found: ID does not exist" containerID="48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9" Oct 31 00:29:03.239205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.238393 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9"} err="failed to get container status \"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9\": rpc error: code = NotFound desc = could not find container \"48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9\": container with ID starting with 48b4d81dac15cab8143295b6cf9a9562f879deb8e32f41aff40e454c753139a9 not found: ID does not exist" Oct 31 00:29:03.239205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.238414 6234 scope.go:122] "RemoveContainer" containerID="ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7" Oct 31 00:29:03.251697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.251673 6234 scope.go:122] "RemoveContainer" containerID="ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7" Oct 31 00:29:03.253181 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:03.253126 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7\": container with ID starting with ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7 not found: ID does not exist" containerID="ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7" Oct 31 00:29:03.253181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.253166 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7"} err="failed to get container status \"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7\": rpc error: code = NotFound desc = could not find container \"ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7\": container with ID starting with ae9c50308fa5b831aaf7f9d64e9dd3d3ad59b7775a326de6d11b4e9b84552fc7 not found: ID does not exist" Oct 31 00:29:03.258194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258163 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.258300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258212 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.258300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258277 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.258430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258316 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.258489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258470 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.258728 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258703 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.258836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258754 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.258908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258836 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.258908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258866 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258922 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.259015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.258976 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259037 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.259122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259058 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.259122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259104 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259145 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.259275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259178 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.259275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259199 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.259428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259301 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259364 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.259428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259386 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259511 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.259587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259519 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\") pod \"277e0d55-a194-44cf-a75f-771a24a9fb65\" (UID: \"277e0d55-a194-44cf-a75f-771a24a9fb65\") " Oct 31 00:29:03.259587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259580 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\") pod \"6fa69024-0355-4478-afa2-ef91dffc4adf\" (UID: \"6fa69024-0355-4478-afa2-ef91dffc4adf\") " Oct 31 00:29:03.259920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259868 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.260012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.259979 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:29:03.260266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260117 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.260350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260294 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260828 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260859 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260875 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260888 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260903 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260917 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260929 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.260943 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.261107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.261074 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:03.261736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.261305 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:03.261816 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.261727 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.262731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.262700 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "kube-api-access-2gftr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.263169 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.263115 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.264468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.264438 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:03.264671 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.264654 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "kube-api-access-xx7l9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.265241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.265224 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:03.265483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.265457 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy" pod "6fa69024-0355-4478-afa2-ef91dffc4adf" (UID: "6fa69024-0355-4478-afa2-ef91dffc4adf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.269235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.269211 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token" pod "277e0d55-a194-44cf-a75f-771a24a9fb65" (UID: "277e0d55-a194-44cf-a75f-771a24a9fb65"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.361828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.361739 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.362028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.361932 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.362028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362015 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.362162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362151 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.362374 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362348 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.362496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362375 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.362612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362518 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.362775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362756 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.362937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362869 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.363394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.363371 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:03.363484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.362904 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.363484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.363472 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.363803 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.363766 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.364144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.364124 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\") pod \"bea6a00f-672b-4a94-971b-106c506294fd\" (UID: \"bea6a00f-672b-4a94-971b-106c506294fd\") " Oct 31 00:29:03.365047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365020 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365052 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365067 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365081 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6fa69024-0355-4478-afa2-ef91dffc4adf-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365095 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2gftr\" (UniqueName: \"kubernetes.io/projected/277e0d55-a194-44cf-a75f-771a24a9fb65-kube-api-access-2gftr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365109 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365122 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/277e0d55-a194-44cf-a75f-771a24a9fb65-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365135 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xx7l9\" (UniqueName: \"kubernetes.io/projected/6fa69024-0355-4478-afa2-ef91dffc4adf-kube-api-access-xx7l9\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365150 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365163 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6fa69024-0355-4478-afa2-ef91dffc4adf-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365175 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365186 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365197 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/bea6a00f-672b-4a94-971b-106c506294fd-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365208 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365224 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.365515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365219 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/277e0d55-a194-44cf-a75f-771a24a9fb65-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.365976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.365705 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "kube-api-access-nlpx6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:03.367510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.367483 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:03.368077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.368054 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo" pod "bea6a00f-672b-4a94-971b-106c506294fd" (UID: "bea6a00f-672b-4a94-971b-106c506294fd"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:03.466696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.466656 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlpx6\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-kube-api-access-nlpx6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.466696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.466690 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/bea6a00f-672b-4a94-971b-106c506294fd-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.466696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.466702 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/bea6a00f-672b-4a94-971b-106c506294fd-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.466955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.466711 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/bea6a00f-672b-4a94-971b-106c506294fd-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:03.529269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.529239 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:29:03.535641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.535609 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-g9llp"] Oct 31 00:29:03.569345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.569302 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:29:03.648312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:03.648215 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-74rzh"] Oct 31 00:29:04.084433 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.084395 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" path="/var/lib/kubelet/pods/277e0d55-a194-44cf-a75f-771a24a9fb65/volumes" Oct 31 00:29:04.085013 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.084993 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" path="/var/lib/kubelet/pods/6fa69024-0355-4478-afa2-ef91dffc4adf/volumes" Oct 31 00:29:04.224593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.224558 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" Oct 31 00:29:04.224593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.224571 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9" event={"ID":"bea6a00f-672b-4a94-971b-106c506294fd","Type":"ContainerDied","Data":"6be2d848741b09295c23b85059499c3381f89e3212983213ab37b0deb7812411"} Oct 31 00:29:04.224864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.224634 6234 scope.go:122] "RemoveContainer" containerID="7d0b112ba9f25716b40cf6c68cacde81b7198d9ff8d2d9c706f4b7232ebb169b" Oct 31 00:29:04.250652 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.250618 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:29:04.257161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.257134 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w1-0df3f8-openshift-default-7c5bbd8947-9tsk9"] Oct 31 00:29:04.487102 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.487068 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:29:04.487305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:04.487279 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" containerID="cri-o://48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a" gracePeriod=30 Oct 31 00:29:06.103488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:06.095941 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bea6a00f-672b-4a94-971b-106c506294fd" path="/var/lib/kubelet/pods/bea6a00f-672b-4a94-971b-106c506294fd/volumes" Oct 31 00:29:09.882312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:09.882280 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:29:10.060051 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060020 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060064 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060126 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060189 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060268 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060299 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060318 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:10.060599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060494 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060501 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:10.060704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060578 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:10.060704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060536 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.060704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060619 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:10.060704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.060693 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\") pod \"03c4be57-3cdc-49d9-8612-069310252339\" (UID: \"03c4be57-3cdc-49d9-8612-069310252339\") " Oct 31 00:29:10.061119 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061096 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:10.061755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061733 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.061891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061763 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.061891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061819 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/03c4be57-3cdc-49d9-8612-069310252339-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.061891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061842 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.061891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.061857 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.062723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.062697 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:10.062723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.062713 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:10.063116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.063097 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:10.063187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.063130 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn" pod "03c4be57-3cdc-49d9-8612-069310252339" (UID: "03c4be57-3cdc-49d9-8612-069310252339"). InnerVolumeSpecName "kube-api-access-xxwnn". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:10.163452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.163416 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/03c4be57-3cdc-49d9-8612-069310252339-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.163452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.163453 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.163645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.163468 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxwnn\" (UniqueName: \"kubernetes.io/projected/03c4be57-3cdc-49d9-8612-069310252339-kube-api-access-xxwnn\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.163645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.163483 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/03c4be57-3cdc-49d9-8612-069310252339-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:10.400622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.400530 6234 generic.go:356] "Generic (PLEG): container finished" podID="03c4be57-3cdc-49d9-8612-069310252339" containerID="48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a" exitCode=0 Oct 31 00:29:10.400895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.400872 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" event={"ID":"03c4be57-3cdc-49d9-8612-069310252339","Type":"ContainerDied","Data":"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a"} Oct 31 00:29:10.401047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.401033 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" event={"ID":"03c4be57-3cdc-49d9-8612-069310252339","Type":"ContainerDied","Data":"d0286bf7cfb7ad60310f2cfdf003e2992741a63127c49f7910410bedc441de4f"} Oct 31 00:29:10.401148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.401137 6234 scope.go:122] "RemoveContainer" containerID="48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a" Oct 31 00:29:10.401380 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.401363 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6" Oct 31 00:29:10.509387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.509349 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:29:10.536863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.536739 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-pg8q6"] Oct 31 00:29:10.616475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.611266 6234 scope.go:122] "RemoveContainer" containerID="48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a" Oct 31 00:29:10.626356 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:10.625142 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a\": container with ID starting with 48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a not found: ID does not exist" containerID="48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a" Oct 31 00:29:10.626356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:10.625185 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a"} err="failed to get container status \"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a\": rpc error: code = NotFound desc = could not find container \"48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a\": container with ID starting with 48d12844feffa5ff0a87e7591c8f349047f5d95ad1364e88826b353cca391a9a not found: ID does not exist" Oct 31 00:29:12.083527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:12.083493 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="03c4be57-3cdc-49d9-8612-069310252339" path="/var/lib/kubelet/pods/03c4be57-3cdc-49d9-8612-069310252339/volumes" Oct 31 00:29:13.240456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.240422 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:29:13.241304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.241250 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" podUID="05f6f790-e53a-49c1-8b17-14c82caf5937" containerName="maas-api" containerID="cri-o://49dd1a506d1b5567cea7edf206447e0d8b15314b3f1d81f4a2de9f1b8acf9cae" gracePeriod=30 Oct 31 00:29:13.442250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.441890 6234 generic.go:356] "Generic (PLEG): container finished" podID="05f6f790-e53a-49c1-8b17-14c82caf5937" containerID="49dd1a506d1b5567cea7edf206447e0d8b15314b3f1d81f4a2de9f1b8acf9cae" exitCode=0 Oct 31 00:29:13.442250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.442172 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" event={"ID":"05f6f790-e53a-49c1-8b17-14c82caf5937","Type":"ContainerDied","Data":"49dd1a506d1b5567cea7edf206447e0d8b15314b3f1d81f4a2de9f1b8acf9cae"} Oct 31 00:29:13.843015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.842980 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:29:13.856157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.856129 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\") pod \"05f6f790-e53a-49c1-8b17-14c82caf5937\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " Oct 31 00:29:13.856296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.856176 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\") pod \"05f6f790-e53a-49c1-8b17-14c82caf5937\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " Oct 31 00:29:13.856296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.856267 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\") pod \"05f6f790-e53a-49c1-8b17-14c82caf5937\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " Oct 31 00:29:13.856439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.856384 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\" (UniqueName: \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\") pod \"05f6f790-e53a-49c1-8b17-14c82caf5937\" (UID: \"05f6f790-e53a-49c1-8b17-14c82caf5937\") " Oct 31 00:29:13.856772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.856742 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca" pod "05f6f790-e53a-49c1-8b17-14c82caf5937" (UID: "05f6f790-e53a-49c1-8b17-14c82caf5937"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:13.857206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.857179 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/05f6f790-e53a-49c1-8b17-14c82caf5937-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.858822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.858768 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls" pod "05f6f790-e53a-49c1-8b17-14c82caf5937" (UID: "05f6f790-e53a-49c1-8b17-14c82caf5937"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:13.859286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.859256 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs" pod "05f6f790-e53a-49c1-8b17-14c82caf5937" (UID: "05f6f790-e53a-49c1-8b17-14c82caf5937"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:13.859376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.859297 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b" pod "05f6f790-e53a-49c1-8b17-14c82caf5937" (UID: "05f6f790-e53a-49c1-8b17-14c82caf5937"). InnerVolumeSpecName "kube-api-access-spl8b". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:13.958749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.958711 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-spl8b\" (UniqueName: \"kubernetes.io/projected/05f6f790-e53a-49c1-8b17-14c82caf5937-kube-api-access-spl8b\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.958749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.958744 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:13.958749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:13.958753 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/05f6f790-e53a-49c1-8b17-14c82caf5937-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:14.450947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:14.450912 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" Oct 31 00:29:14.451443 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:14.450947 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs" event={"ID":"05f6f790-e53a-49c1-8b17-14c82caf5937","Type":"ContainerDied","Data":"c4ceec8e3cbc09559fdc3c22382597a46b8b4e8918ea3c3756284079eb13e3d8"} Oct 31 00:29:14.451443 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:14.451011 6234 scope.go:122] "RemoveContainer" containerID="49dd1a506d1b5567cea7edf206447e0d8b15314b3f1d81f4a2de9f1b8acf9cae" Oct 31 00:29:14.477219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:14.477187 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:29:14.480769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:14.480741 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-d609e2-c96c7d5d7-mkjrs"] Oct 31 00:29:15.755638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:15.755600 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:29:15.756159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:15.755922 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" podUID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" containerName="maas-api" containerID="cri-o://5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e" gracePeriod=30 Oct 31 00:29:16.128744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.128705 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05f6f790-e53a-49c1-8b17-14c82caf5937" path="/var/lib/kubelet/pods/05f6f790-e53a-49c1-8b17-14c82caf5937/volumes" Oct 31 00:29:16.156484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.156454 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:29:16.315612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.315502 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\") pod \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " Oct 31 00:29:16.315839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.315636 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\") pod \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " Oct 31 00:29:16.315839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.315684 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\") pod \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " Oct 31 00:29:16.315839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.315761 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\" (UniqueName: \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\") pod \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\" (UID: \"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1\") " Oct 31 00:29:16.321059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.317374 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca" pod "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" (UID: "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:16.338826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.337267 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs" pod "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" (UID: "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:16.338826 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.338413 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57" pod "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" (UID: "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1"). InnerVolumeSpecName "kube-api-access-2sw57". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:16.339661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.339633 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls" pod "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" (UID: "88b26cc5-6eaf-4a46-8197-f63ffce6d8d1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:16.422812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.420276 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:16.422812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.420335 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:16.422812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.420350 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2sw57\" (UniqueName: \"kubernetes.io/projected/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-kube-api-access-2sw57\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:16.422812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.420364 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:16.477606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.476483 6234 generic.go:356] "Generic (PLEG): container finished" podID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" containerID="5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e" exitCode=0 Oct 31 00:29:16.477606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.476544 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" Oct 31 00:29:16.477606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.476662 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" event={"ID":"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1","Type":"ContainerDied","Data":"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e"} Oct 31 00:29:16.477606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.476708 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n" event={"ID":"88b26cc5-6eaf-4a46-8197-f63ffce6d8d1","Type":"ContainerDied","Data":"439c40c22bc6eb3c3bb0f41822854c2d1ffd37b177b88c506531878349e0f965"} Oct 31 00:29:16.477606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.476739 6234 scope.go:122] "RemoveContainer" containerID="5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e" Oct 31 00:29:16.489338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.489316 6234 scope.go:122] "RemoveContainer" containerID="5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e" Oct 31 00:29:16.489651 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:16.489623 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e\": container with ID starting with 5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e not found: ID does not exist" containerID="5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e" Oct 31 00:29:16.489710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.489664 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e"} err="failed to get container status \"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e\": rpc error: code = NotFound desc = could not find container \"5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e\": container with ID starting with 5790c83c4b687676e3f7233bf1fa8d793566be2d1dd96b091f63652c01b2ee3e not found: ID does not exist" Oct 31 00:29:16.500889 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.499003 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:29:16.512338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.512307 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:29:16.516328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.516303 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w1-7a7688-58d99b7cc-vr54n"] Oct 31 00:29:16.522505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:16.522479 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:29:18.085911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:18.085828 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" path="/var/lib/kubelet/pods/88b26cc5-6eaf-4a46-8197-f63ffce6d8d1/volumes" Oct 31 00:29:21.499371 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:21.499296 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" podUID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" containerName="payload-processing" containerID="cri-o://4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59" gracePeriod=25 Oct 31 00:29:21.523186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:21.523156 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" podUID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" containerName="payload-pre-processing" containerID="cri-o://a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d" gracePeriod=25 Oct 31 00:29:21.609925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:21.609887 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:29:21.610234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:21.610172 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" containerID="cri-o://511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77" gracePeriod=30 Oct 31 00:29:22.037042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.037009 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:29:22.077577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.077501 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\" (UniqueName: \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\") pod \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " Oct 31 00:29:22.077829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.077749 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\") pod \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\" (UID: \"3486bbdd-f413-4abb-9f7a-50d56787a7f6\") " Oct 31 00:29:22.078488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.078460 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume" pod "3486bbdd-f413-4abb-9f7a-50d56787a7f6" (UID: "3486bbdd-f413-4abb-9f7a-50d56787a7f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:22.079036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.078976 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3486bbdd-f413-4abb-9f7a-50d56787a7f6-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:22.080607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.080583 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m" pod "3486bbdd-f413-4abb-9f7a-50d56787a7f6" (UID: "3486bbdd-f413-4abb-9f7a-50d56787a7f6"). InnerVolumeSpecName "kube-api-access-lv45m". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:22.114300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.114278 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:29:22.180092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.180053 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\" (UniqueName: \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\") pod \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " Oct 31 00:29:22.180478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.180455 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\") pod \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\" (UID: \"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9\") " Oct 31 00:29:22.180966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.180936 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume" pod "9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" (UID: "9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:22.181206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.181186 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lv45m\" (UniqueName: \"kubernetes.io/projected/3486bbdd-f413-4abb-9f7a-50d56787a7f6-kube-api-access-lv45m\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:22.181290 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.181214 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:22.182333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.182311 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp" pod "9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" (UID: "9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9"). InnerVolumeSpecName "kube-api-access-dt6lp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:22.282521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.282440 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dt6lp\" (UniqueName: \"kubernetes.io/projected/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9-kube-api-access-dt6lp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:22.526858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.526830 6234 generic.go:356] "Generic (PLEG): container finished" podID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" containerID="a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d" exitCode=0 Oct 31 00:29:22.527300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.526873 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" Oct 31 00:29:22.527300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.526918 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" event={"ID":"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9","Type":"ContainerDied","Data":"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d"} Oct 31 00:29:22.527300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.526982 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg" event={"ID":"9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9","Type":"ContainerDied","Data":"9d849522dc87418ac255448a8dc9d73b9a36f32f107b884a6ba536cb35223797"} Oct 31 00:29:22.527300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.527015 6234 scope.go:122] "RemoveContainer" containerID="a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d" Oct 31 00:29:22.528553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.528536 6234 generic.go:356] "Generic (PLEG): container finished" podID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" containerID="4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59" exitCode=0 Oct 31 00:29:22.528654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.528589 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" Oct 31 00:29:22.528654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.528622 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" event={"ID":"3486bbdd-f413-4abb-9f7a-50d56787a7f6","Type":"ContainerDied","Data":"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59"} Oct 31 00:29:22.528749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.528668 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x" event={"ID":"3486bbdd-f413-4abb-9f7a-50d56787a7f6","Type":"ContainerDied","Data":"c75810fe4f8b622b48e729b3953429da741bd0d3f0e1a232d59914a1ddede594"} Oct 31 00:29:22.540151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.540135 6234 scope.go:122] "RemoveContainer" containerID="a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d" Oct 31 00:29:22.540455 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:22.540436 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d\": container with ID starting with a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d not found: ID does not exist" containerID="a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d" Oct 31 00:29:22.540541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.540466 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d"} err="failed to get container status \"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d\": rpc error: code = NotFound desc = could not find container \"a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d\": container with ID starting with a3085338bba8f40b0ff827ab0e97b893f178b921ecebc25e8eb18ffb75b6d29d not found: ID does not exist" Oct 31 00:29:22.540541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.540488 6234 scope.go:122] "RemoveContainer" containerID="4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59" Oct 31 00:29:22.552664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.552641 6234 scope.go:122] "RemoveContainer" containerID="4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59" Oct 31 00:29:22.552998 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:22.552978 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59\": container with ID starting with 4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59 not found: ID does not exist" containerID="4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59" Oct 31 00:29:22.553076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.553004 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59"} err="failed to get container status \"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59\": rpc error: code = NotFound desc = could not find container \"4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59\": container with ID starting with 4da653fcf3dd843600f821f873bb9b34777d89c7cf8fa2b9f94342362c478b59 not found: ID does not exist" Oct 31 00:29:22.553136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.553090 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:29:22.557600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.557580 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-41a0b7-769fd46cc9-28n8x"] Oct 31 00:29:22.568934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.568915 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:29:22.574750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:22.574729 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-41a0b7-59dc7cfff5-x7lsg"] Oct 31 00:29:24.083951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:24.083918 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" path="/var/lib/kubelet/pods/3486bbdd-f413-4abb-9f7a-50d56787a7f6/volumes" Oct 31 00:29:24.084483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:24.084467 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" path="/var/lib/kubelet/pods/9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9/volumes" Oct 31 00:29:26.272351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.272313 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:29:26.272745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.272604 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" containerID="cri-o://21b1b766ab017aa0de9846f91f44eab0ae9333994c7c2b38244c127cf8c00ec2" gracePeriod=30 Oct 31 00:29:26.298226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.298194 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:29:26.298405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.298387 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" containerID="cri-o://fe9b29b888031720e605ff106b29e4122ed1ef267a002b46770377412b4c9c92" gracePeriod=30 Oct 31 00:29:26.306464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.306424 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:29:26.306677 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.306654 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" containerID="cri-o://81a7c8b8fbf3008a7d03ea62f077c220df3dba4da8666f8646cd90dfd5c0e03a" gracePeriod=30 Oct 31 00:29:26.728010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.727976 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:26.728141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.728037 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:26.796878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.796011 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:26.796878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.796075 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:26.961192 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:26.961163 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:29:27.013144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013112 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013164 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013239 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013288 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013326 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013448 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:27.013601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013478 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013729 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013711 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:27.013819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013747 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013743 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:27.013956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013868 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.013956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.013900 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\") pod \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\" (UID: \"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9\") " Oct 31 00:29:27.014534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.014506 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:27.014768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.014744 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.014884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.014773 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.014884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.014809 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.014884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.014825 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.015317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.015290 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:27.017315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.017287 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:27.017435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.017326 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:27.017435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.017424 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:27.017617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.017596 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf" pod "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" (UID: "3ac6ba56-d1b7-44fc-b8ce-4125405a30a9"). InnerVolumeSpecName "kube-api-access-qxcwf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:27.116459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.116429 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxcwf\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-kube-api-access-qxcwf\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.116459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.116458 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.116459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.116467 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.116705 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.116477 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.116705 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.116486 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:27.589059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.589021 6234 generic.go:356] "Generic (PLEG): container finished" podID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerID="511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77" exitCode=0 Oct 31 00:29:27.589484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.589063 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" Oct 31 00:29:27.589484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.589093 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerDied","Data":"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77"} Oct 31 00:29:27.589484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.589161 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz" event={"ID":"3ac6ba56-d1b7-44fc-b8ce-4125405a30a9","Type":"ContainerDied","Data":"e4fe8461eaa61da464b3450656b0f22e215b0d2d8267d872431625ad92e10448"} Oct 31 00:29:27.589484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.589186 6234 scope.go:122] "RemoveContainer" containerID="511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77" Oct 31 00:29:27.618517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.618493 6234 scope.go:122] "RemoveContainer" containerID="24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1" Oct 31 00:29:27.620422 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.620398 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:29:27.626286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.626259 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-d609e2-openshift-default-7c78d56c57-nsdwz"] Oct 31 00:29:27.630516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.630494 6234 scope.go:122] "RemoveContainer" containerID="511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77" Oct 31 00:29:27.630866 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:27.630845 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77\": container with ID starting with 511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77 not found: ID does not exist" containerID="511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77" Oct 31 00:29:27.630933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.630876 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77"} err="failed to get container status \"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77\": rpc error: code = NotFound desc = could not find container \"511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77\": container with ID starting with 511b54f7ba08cb388a1211f1de9efdab149810a15ffaabe7523970a20cec7e77 not found: ID does not exist" Oct 31 00:29:27.630933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.630897 6234 scope.go:122] "RemoveContainer" containerID="24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1" Oct 31 00:29:27.631195 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:27.631164 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1\": container with ID starting with 24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1 not found: ID does not exist" containerID="24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1" Oct 31 00:29:27.631274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:27.631205 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1"} err="failed to get container status \"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1\": rpc error: code = NotFound desc = could not find container \"24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1\": container with ID starting with 24bf744db14cf9e81ce74b9d776c458d524225fc11ec6bcf4242fb7b1095daa1 not found: ID does not exist" Oct 31 00:29:28.085394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:28.085363 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" path="/var/lib/kubelet/pods/3ac6ba56-d1b7-44fc-b8ce-4125405a30a9/volumes" Oct 31 00:29:31.748749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.748717 6234 generic.go:356] "Generic (PLEG): container finished" podID="0f189999-8354-4304-8558-d88276d509cf" containerID="21b1b766ab017aa0de9846f91f44eab0ae9333994c7c2b38244c127cf8c00ec2" exitCode=0 Oct 31 00:29:31.749367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.749344 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" event={"ID":"0f189999-8354-4304-8558-d88276d509cf","Type":"ContainerDied","Data":"21b1b766ab017aa0de9846f91f44eab0ae9333994c7c2b38244c127cf8c00ec2"} Oct 31 00:29:31.752376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.752354 6234 generic.go:356] "Generic (PLEG): container finished" podID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerID="81a7c8b8fbf3008a7d03ea62f077c220df3dba4da8666f8646cd90dfd5c0e03a" exitCode=0 Oct 31 00:29:31.752838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.752575 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerDied","Data":"81a7c8b8fbf3008a7d03ea62f077c220df3dba4da8666f8646cd90dfd5c0e03a"} Oct 31 00:29:31.756192 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.756001 6234 generic.go:356] "Generic (PLEG): container finished" podID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerID="fe9b29b888031720e605ff106b29e4122ed1ef267a002b46770377412b4c9c92" exitCode=0 Oct 31 00:29:31.756342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.756325 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerDied","Data":"fe9b29b888031720e605ff106b29e4122ed1ef267a002b46770377412b4c9c92"} Oct 31 00:29:31.756502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.756448 6234 scope.go:122] "RemoveContainer" containerID="409155e20dfe886c7542b5c49a5dda7059ba4906d19225f5c22b8d3761bf7c13" Oct 31 00:29:31.774262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.773921 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:29:31.782297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.782197 6234 scope.go:122] "RemoveContainer" containerID="a9bdb87265075b8eeb894343911dc25c2b8f9ddf37e2694fe584b719ebee327a" Oct 31 00:29:31.796736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.796703 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.796958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.796938 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.797282 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.797263 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.797436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.797418 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.797699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.797679 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.798484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.798449 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.798673 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.798654 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.799399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.799377 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.799544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.799528 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\") pod \"0f189999-8354-4304-8558-d88276d509cf\" (UID: \"0f189999-8354-4304-8558-d88276d509cf\") " Oct 31 00:29:31.800574 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.798079 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.805613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.800952 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.805742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.801232 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:31.805915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.802483 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.806011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.803256 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.818299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.818264 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:29:31.827255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.826961 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:31.831581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.831045 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:31.831581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.831095 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.831581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.831334 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj" pod "0f189999-8354-4304-8558-d88276d509cf" (UID: "0f189999-8354-4304-8558-d88276d509cf"). InnerVolumeSpecName "kube-api-access-dchjj". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:31.911397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.911273 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.911772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.911749 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.911966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.911948 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.912052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.912042 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.912257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.912240 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f189999-8354-4304-8558-d88276d509cf-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.912365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.912351 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.913765 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.913744 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f189999-8354-4304-8558-d88276d509cf-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.913947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.913929 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f189999-8354-4304-8558-d88276d509cf-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:31.914063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:31.914037 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dchjj\" (UniqueName: \"kubernetes.io/projected/0f189999-8354-4304-8558-d88276d509cf-kube-api-access-dchjj\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.085367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.085336 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:29:32.098068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.098042 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190295 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190375 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190433 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190474 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190520 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190550 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190640 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190702 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190734 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190775 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190905 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.190966 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191001 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191105 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191186 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191256 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191334 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\") pod \"4229234f-54c4-40bd-804f-6ce0436a15a0\" (UID: \"4229234f-54c4-40bd-804f-6ce0436a15a0\") " Oct 31 00:29:32.204810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.191366 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\") pod \"07b588a2-f55d-4fcc-bf3f-db6834789de6\" (UID: \"07b588a2-f55d-4fcc-bf3f-db6834789de6\") " Oct 31 00:29:32.205911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.205528 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.205911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.205873 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.220832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.209863 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.220832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.218377 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.220832 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.219297 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.229814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.221259 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:32.230525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.230498 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.588844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.466324 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:32.694226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694184 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:32.695905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694194 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "kube-api-access-x9m9p". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:32.696059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694528 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:32.696147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694580 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:32.696234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694629 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:32.696311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.694678 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.703186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.702874 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.704040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.703361 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket" pod "4229234f-54c4-40bd-804f-6ce0436a15a0" (UID: "4229234f-54c4-40bd-804f-6ce0436a15a0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.704040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.704000 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:32.764105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.764069 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.779548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.779519 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.866810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.828942 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.867059 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.867033 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4229234f-54c4-40bd-804f-6ce0436a15a0-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.867201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.867186 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.939809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.937193 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth" pod "07b588a2-f55d-4fcc-bf3f-db6834789de6" (UID: "07b588a2-f55d-4fcc-bf3f-db6834789de6"). InnerVolumeSpecName "kube-api-access-8wwth". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.867301 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954532 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954576 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07b588a2-f55d-4fcc-bf3f-db6834789de6-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954592 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954608 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954640 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954656 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07b588a2-f55d-4fcc-bf3f-db6834789de6-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954671 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954689 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x9m9p\" (UniqueName: \"kubernetes.io/projected/4229234f-54c4-40bd-804f-6ce0436a15a0-kube-api-access-x9m9p\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954723 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954738 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07b588a2-f55d-4fcc-bf3f-db6834789de6-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.963131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.954752 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4229234f-54c4-40bd-804f-6ce0436a15a0-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:32.996820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.996728 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" event={"ID":"07b588a2-f55d-4fcc-bf3f-db6834789de6","Type":"ContainerDied","Data":"27e9e4711ac97008636cb653fc2405f2f5926a22bdb55b64a8ec21c7d83c2227"} Oct 31 00:29:32.997019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.996869 6234 scope.go:122] "RemoveContainer" containerID="fe9b29b888031720e605ff106b29e4122ed1ef267a002b46770377412b4c9c92" Oct 31 00:29:32.997092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:32.997054 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm" Oct 31 00:29:33.092824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.060879 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" event={"ID":"0f189999-8354-4304-8558-d88276d509cf","Type":"ContainerDied","Data":"638e06d4137157a04a24bb6d05722cf6d23e0c2e734641ff47a2ccb85d646f2f"} Oct 31 00:29:33.092824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.061026 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf" Oct 31 00:29:33.374597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.374260 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" probeResult="failure" output="Get \"https://10.134.0.72:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:33.428817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.413916 6234 scope.go:122] "RemoveContainer" containerID="21b1b766ab017aa0de9846f91f44eab0ae9333994c7c2b38244c127cf8c00ec2" Oct 31 00:29:33.436942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.436905 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8wwth\" (UniqueName: \"kubernetes.io/projected/07b588a2-f55d-4fcc-bf3f-db6834789de6-kube-api-access-8wwth\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:33.554805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.512139 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" probeResult="failure" output="Get \"https://10.134.0.69:8000/ready\": context deadline exceeded" Oct 31 00:29:33.644933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.633907 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:29:33.649807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.646310 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": context deadline exceeded" Oct 31 00:29:33.657810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.655081 6234 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:33.740548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.740514 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" event={"ID":"4229234f-54c4-40bd-804f-6ce0436a15a0","Type":"ContainerDied","Data":"37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13"} Oct 31 00:29:33.740930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.740906 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f" Oct 31 00:29:33.879999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.879970 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:29:33.883759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.883733 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:29:33.892033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.891997 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-2szvf"] Oct 31 00:29:33.916122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.916095 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:29:33.918057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.918037 6234 scope.go:122] "RemoveContainer" containerID="81a7c8b8fbf3008a7d03ea62f077c220df3dba4da8666f8646cd90dfd5c0e03a" Oct 31 00:29:33.923400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:33.923380 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-6l67f"] Oct 31 00:29:34.025865 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.025824 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:29:34.026325 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:34.026299 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:29:34.043902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.043864 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" probeResult="failure" output="Get \"https://10.134.0.68:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" Oct 31 00:29:34.077963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.077931 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w1-7a7688-openshift-default-7dfd656469-52tkm"] Oct 31 00:29:34.194894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.193232 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" path="/var/lib/kubelet/pods/07b588a2-f55d-4fcc-bf3f-db6834789de6/volumes" Oct 31 00:29:34.194894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.194042 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f189999-8354-4304-8558-d88276d509cf" path="/var/lib/kubelet/pods/0f189999-8354-4304-8558-d88276d509cf/volumes" Oct 31 00:29:34.249163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:34.237458 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" path="/var/lib/kubelet/pods/4229234f-54c4-40bd-804f-6ce0436a15a0/volumes" Oct 31 00:29:34.317455 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:34.317414 6234 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4229234f_54c4_40bd_804f_6ce0436a15a0.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07b588a2_f55d_4fcc_bf3f_db6834789de6.slice\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4229234f_54c4_40bd_804f_6ce0436a15a0.slice/crio-37c264ac01dcb222e8773e794f332c61d414640a9bd8879b50f334716159db13\": RecentStats: unable to find data in memory cache]" Oct 31 00:29:37.149464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:37.149432 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:29:37.167622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:37.167588 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:29:38.635238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:38.635194 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" podUID="6fa3b215-6d76-4d82-8f57-f23484ecf880" containerName="payload-processing" containerID="cri-o://0cc5fcc1de68fddc1c0490080b47023879a82ddfd14b8ce5803981e44431a70b" gracePeriod=25 Oct 31 00:29:38.785333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:38.785294 6234 generic.go:356] "Generic (PLEG): container finished" podID="6fa3b215-6d76-4d82-8f57-f23484ecf880" containerID="0cc5fcc1de68fddc1c0490080b47023879a82ddfd14b8ce5803981e44431a70b" exitCode=0 Oct 31 00:29:38.785516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:38.785368 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" event={"ID":"6fa3b215-6d76-4d82-8f57-f23484ecf880","Type":"ContainerDied","Data":"0cc5fcc1de68fddc1c0490080b47023879a82ddfd14b8ce5803981e44431a70b"} Oct 31 00:29:38.881125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:38.881089 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerName="payload-pre-processing" containerID="cri-o://5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2" gracePeriod=25 Oct 31 00:29:38.976654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:38.976626 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:29:39.056183 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:29:39.056157 6234 logging.go:55] [core] [Channel #4249 SubChannel #4250]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.148:50052", ServerName: "10.134.0.148:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.148:50052: connect: connection refused" Oct 31 00:29:39.083769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.083730 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\") pod \"6fa3b215-6d76-4d82-8f57-f23484ecf880\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " Oct 31 00:29:39.084182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.084150 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\" (UniqueName: \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\") pod \"6fa3b215-6d76-4d82-8f57-f23484ecf880\" (UID: \"6fa3b215-6d76-4d82-8f57-f23484ecf880\") " Oct 31 00:29:39.084283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.084174 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume" pod "6fa3b215-6d76-4d82-8f57-f23484ecf880" (UID: "6fa3b215-6d76-4d82-8f57-f23484ecf880"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:39.086845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.086814 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6fa3b215-6d76-4d82-8f57-f23484ecf880-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:39.105866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.105812 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx" pod "6fa3b215-6d76-4d82-8f57-f23484ecf880" (UID: "6fa3b215-6d76-4d82-8f57-f23484ecf880"). InnerVolumeSpecName "kube-api-access-vj5xx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:39.162272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.162246 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:29:39.188100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.188066 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\") pod \"50ff4ab8-21fa-415c-a775-f56c3fca9112\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " Oct 31 00:29:39.188265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.188139 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\" (UniqueName: \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\") pod \"50ff4ab8-21fa-415c-a775-f56c3fca9112\" (UID: \"50ff4ab8-21fa-415c-a775-f56c3fca9112\") " Oct 31 00:29:39.188499 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.188472 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume" pod "50ff4ab8-21fa-415c-a775-f56c3fca9112" (UID: "50ff4ab8-21fa-415c-a775-f56c3fca9112"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:39.188974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.188947 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/50ff4ab8-21fa-415c-a775-f56c3fca9112-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:39.189123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.188978 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj5xx\" (UniqueName: \"kubernetes.io/projected/6fa3b215-6d76-4d82-8f57-f23484ecf880-kube-api-access-vj5xx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:39.191087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.191064 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6" pod "50ff4ab8-21fa-415c-a775-f56c3fca9112" (UID: "50ff4ab8-21fa-415c-a775-f56c3fca9112"). InnerVolumeSpecName "kube-api-access-2p5r6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:39.290588 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.290554 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2p5r6\" (UniqueName: \"kubernetes.io/projected/50ff4ab8-21fa-415c-a775-f56c3fca9112-kube-api-access-2p5r6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:39.795324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.795291 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" Oct 31 00:29:39.795774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.795290 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4" event={"ID":"6fa3b215-6d76-4d82-8f57-f23484ecf880","Type":"ContainerDied","Data":"8f57e569aa1d8f8036c214c08e021eca96b7e11e22f5215e8dae77f16e7b3f49"} Oct 31 00:29:39.795774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.795475 6234 scope.go:122] "RemoveContainer" containerID="0cc5fcc1de68fddc1c0490080b47023879a82ddfd14b8ce5803981e44431a70b" Oct 31 00:29:39.796893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.796817 6234 generic.go:356] "Generic (PLEG): container finished" podID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerID="5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2" exitCode=0 Oct 31 00:29:39.796893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.796857 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" event={"ID":"50ff4ab8-21fa-415c-a775-f56c3fca9112","Type":"ContainerDied","Data":"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2"} Oct 31 00:29:39.796893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.796880 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" Oct 31 00:29:39.797061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.796895 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" event={"ID":"50ff4ab8-21fa-415c-a775-f56c3fca9112","Type":"ContainerDied","Data":"b871f8006af85395702500f1914fa171fb77cb540db98be4ceeb80bb360ede64"} Oct 31 00:29:39.808539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.808521 6234 scope.go:122] "RemoveContainer" containerID="5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2" Oct 31 00:29:39.818654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.818635 6234 scope.go:122] "RemoveContainer" containerID="5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2" Oct 31 00:29:39.819063 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:39.819042 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2\": container with ID starting with 5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2 not found: ID does not exist" containerID="5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2" Oct 31 00:29:39.819159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.819071 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2"} err="failed to get container status \"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2\": rpc error: code = NotFound desc = could not find container \"5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2\": container with ID starting with 5d54749117a480d16ca8c33e72f397a4ce4a423c121882f14b1f50b89f58c0b2 not found: ID does not exist" Oct 31 00:29:39.827527 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.827508 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:29:39.833314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.833289 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d898f3-57987b846f-ndgz4"] Oct 31 00:29:39.848856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.848832 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:29:39.852434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:39.852410 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj"] Oct 31 00:29:40.002880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:40.002850 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:29:40.056600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:40.055241 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-api-b-d898f3-688b4f49f9-2lhwj" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.148:50052\" within 1s: context deadline exceeded" Oct 31 00:29:40.089595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:40.089558 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" path="/var/lib/kubelet/pods/50ff4ab8-21fa-415c-a775-f56c3fca9112/volumes" Oct 31 00:29:40.090311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:40.090290 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fa3b215-6d76-4d82-8f57-f23484ecf880" path="/var/lib/kubelet/pods/6fa3b215-6d76-4d82-8f57-f23484ecf880/volumes" Oct 31 00:29:42.150211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:42.150162 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" containerID="cri-o://9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c" gracePeriod=25 Oct 31 00:29:42.168850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:42.168816 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" containerID="cri-o://df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d" gracePeriod=25 Oct 31 00:29:43.693684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.693637 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:29:43.694205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.693919 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" podUID="75097016-cdfc-4218-8c24-bee5d1bc31a1" containerName="maas-api" containerID="cri-o://406328ccaba2fe99a9f54f7066948a30340d888e72e0891fd7e8820c4dd2a932" gracePeriod=30 Oct 31 00:29:43.829854 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.829808 6234 generic.go:356] "Generic (PLEG): container finished" podID="75097016-cdfc-4218-8c24-bee5d1bc31a1" containerID="406328ccaba2fe99a9f54f7066948a30340d888e72e0891fd7e8820c4dd2a932" exitCode=0 Oct 31 00:29:43.830018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.829812 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" event={"ID":"75097016-cdfc-4218-8c24-bee5d1bc31a1","Type":"ContainerDied","Data":"406328ccaba2fe99a9f54f7066948a30340d888e72e0891fd7e8820c4dd2a932"} Oct 31 00:29:43.883262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.882957 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:29:43.883948 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:43.883902 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" podUID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" containerName="maas-api" containerID="cri-o://5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1" gracePeriod=30 Oct 31 00:29:44.021571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.021539 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:29:44.092504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.092472 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\" (UniqueName: \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\") pod \"75097016-cdfc-4218-8c24-bee5d1bc31a1\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " Oct 31 00:29:44.092645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.092578 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\") pod \"75097016-cdfc-4218-8c24-bee5d1bc31a1\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " Oct 31 00:29:44.092645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.092621 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\") pod \"75097016-cdfc-4218-8c24-bee5d1bc31a1\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " Oct 31 00:29:44.092767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.092731 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\") pod \"75097016-cdfc-4218-8c24-bee5d1bc31a1\" (UID: \"75097016-cdfc-4218-8c24-bee5d1bc31a1\") " Oct 31 00:29:44.093503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.093457 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca" pod "75097016-cdfc-4218-8c24-bee5d1bc31a1" (UID: "75097016-cdfc-4218-8c24-bee5d1bc31a1"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:44.108757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.108722 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls" pod "75097016-cdfc-4218-8c24-bee5d1bc31a1" (UID: "75097016-cdfc-4218-8c24-bee5d1bc31a1"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:44.108757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.108750 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp" pod "75097016-cdfc-4218-8c24-bee5d1bc31a1" (UID: "75097016-cdfc-4218-8c24-bee5d1bc31a1"). InnerVolumeSpecName "kube-api-access-7tfpp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:44.113198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.112241 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs" pod "75097016-cdfc-4218-8c24-bee5d1bc31a1" (UID: "75097016-cdfc-4218-8c24-bee5d1bc31a1"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:44.246907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.199497 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.246907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.199543 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/75097016-cdfc-4218-8c24-bee5d1bc31a1-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.246907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.199558 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/75097016-cdfc-4218-8c24-bee5d1bc31a1-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.246907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.199574 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7tfpp\" (UniqueName: \"kubernetes.io/projected/75097016-cdfc-4218-8c24-bee5d1bc31a1-kube-api-access-7tfpp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.349815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.345999 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:29:44.415219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.415180 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\") pod \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " Oct 31 00:29:44.415406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.415379 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\" (UniqueName: \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\") pod \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " Oct 31 00:29:44.415455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.415416 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\") pod \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " Oct 31 00:29:44.415455 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.415446 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\") pod \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\" (UID: \"d3aa20d4-7be0-48fb-9b28-c18b47f89b15\") " Oct 31 00:29:44.417924 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.417898 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca" pod "d3aa20d4-7be0-48fb-9b28-c18b47f89b15" (UID: "d3aa20d4-7be0-48fb-9b28-c18b47f89b15"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:44.419453 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.419426 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq" pod "d3aa20d4-7be0-48fb-9b28-c18b47f89b15" (UID: "d3aa20d4-7be0-48fb-9b28-c18b47f89b15"). InnerVolumeSpecName "kube-api-access-f9vmq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:44.420111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.420089 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs" pod "d3aa20d4-7be0-48fb-9b28-c18b47f89b15" (UID: "d3aa20d4-7be0-48fb-9b28-c18b47f89b15"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:44.421031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.420998 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls" pod "d3aa20d4-7be0-48fb-9b28-c18b47f89b15" (UID: "d3aa20d4-7be0-48fb-9b28-c18b47f89b15"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:44.518838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.518712 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.519003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.518865 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f9vmq\" (UniqueName: \"kubernetes.io/projected/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-kube-api-access-f9vmq\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.519003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.518894 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.519003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.518942 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/d3aa20d4-7be0-48fb-9b28-c18b47f89b15-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:44.840408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.840316 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" Oct 31 00:29:44.840408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.840338 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js" event={"ID":"75097016-cdfc-4218-8c24-bee5d1bc31a1","Type":"ContainerDied","Data":"e19a67e4a617996ebe26e6ead9a08f7abcab5ce1c724f6252447bcb56ca70707"} Oct 31 00:29:44.840896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.840418 6234 scope.go:122] "RemoveContainer" containerID="406328ccaba2fe99a9f54f7066948a30340d888e72e0891fd7e8820c4dd2a932" Oct 31 00:29:44.842111 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.842084 6234 generic.go:356] "Generic (PLEG): container finished" podID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" containerID="5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1" exitCode=0 Oct 31 00:29:44.842265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.842140 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" Oct 31 00:29:44.842265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.842188 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" event={"ID":"d3aa20d4-7be0-48fb-9b28-c18b47f89b15","Type":"ContainerDied","Data":"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1"} Oct 31 00:29:44.842265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.842234 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v" event={"ID":"d3aa20d4-7be0-48fb-9b28-c18b47f89b15","Type":"ContainerDied","Data":"3cbff5597066b6ef3fb18ed7b5ee74cfc2dd0aecf1611f0059f3a8883eaeba57"} Oct 31 00:29:44.852496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.852466 6234 scope.go:122] "RemoveContainer" containerID="5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1" Oct 31 00:29:44.862489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.862467 6234 scope.go:122] "RemoveContainer" containerID="5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1" Oct 31 00:29:44.862881 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:44.862860 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1\": container with ID starting with 5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1 not found: ID does not exist" containerID="5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1" Oct 31 00:29:44.862936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.862891 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1"} err="failed to get container status \"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1\": rpc error: code = NotFound desc = could not find container \"5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1\": container with ID starting with 5e2cba2cf7c27a9b94ad03581f4088547587368e42b03bcaaf0945f595a73cb1 not found: ID does not exist" Oct 31 00:29:44.910107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.910076 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:29:44.920123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.920089 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-a39be4-74d8c99855-s97js"] Oct 31 00:29:44.935744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.935714 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:29:44.943464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:44.943435 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-cc0fe1-68bb48d99f-w6g4v"] Oct 31 00:29:45.003275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.003218 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" podUID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" containerName="payload-processing" containerID="cri-o://c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424" gracePeriod=25 Oct 31 00:29:45.113364 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.113280 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:29:45.113513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.113291 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:29:45.281576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.281549 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:29:45.332430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.332389 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\" (UniqueName: \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\") pod \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " Oct 31 00:29:45.334736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.334706 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc" pod "9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" (UID: "9546dbe9-0f87-4b6a-bc1a-3f5440c4b629"). InnerVolumeSpecName "kube-api-access-8l2nc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:45.434094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.433997 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\") pod \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\" (UID: \"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629\") " Oct 31 00:29:45.434429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.434405 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume" pod "9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" (UID: "9546dbe9-0f87-4b6a-bc1a-3f5440c4b629"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:45.434580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.434565 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:45.434664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.434581 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8l2nc\" (UniqueName: \"kubernetes.io/projected/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629-kube-api-access-8l2nc\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:45.852117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.852086 6234 generic.go:356] "Generic (PLEG): container finished" podID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" containerID="c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424" exitCode=0 Oct 31 00:29:45.852594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.852141 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" Oct 31 00:29:45.852594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.852173 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" event={"ID":"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629","Type":"ContainerDied","Data":"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424"} Oct 31 00:29:45.852594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.852226 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl" event={"ID":"9546dbe9-0f87-4b6a-bc1a-3f5440c4b629","Type":"ContainerDied","Data":"eb7577998c8bfe28537901269adb0fed1befb6f9bd3d20e13f3187ab957b3150"} Oct 31 00:29:45.852594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.852257 6234 scope.go:122] "RemoveContainer" containerID="c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424" Oct 31 00:29:45.862536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.862519 6234 scope.go:122] "RemoveContainer" containerID="c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424" Oct 31 00:29:45.862812 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:45.862765 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424\": container with ID starting with c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424 not found: ID does not exist" containerID="c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424" Oct 31 00:29:45.862887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.862869 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424"} err="failed to get container status \"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424\": rpc error: code = NotFound desc = could not find container \"c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424\": container with ID starting with c627fff8a9a02ae435b52e25c30ef1050075e1f2d497b16a0cbd7d3229967424 not found: ID does not exist" Oct 31 00:29:45.879354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.879328 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:29:45.885047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:45.885024 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w4-a39be4-7fd9f88b79-zs5xl"] Oct 31 00:29:46.083964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:46.083931 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75097016-cdfc-4218-8c24-bee5d1bc31a1" path="/var/lib/kubelet/pods/75097016-cdfc-4218-8c24-bee5d1bc31a1/volumes" Oct 31 00:29:46.084467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:46.084447 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" path="/var/lib/kubelet/pods/9546dbe9-0f87-4b6a-bc1a-3f5440c4b629/volumes" Oct 31 00:29:46.084960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:46.084942 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" path="/var/lib/kubelet/pods/d3aa20d4-7be0-48fb-9b28-c18b47f89b15/volumes" Oct 31 00:29:46.940229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:46.940165 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" containerID="cri-o://9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765" gracePeriod=45 Oct 31 00:29:47.210431 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.210364 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:29:47.371295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371262 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\" (UniqueName: \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371314 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371359 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371413 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\" (UniqueName: \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371597 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.371708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371657 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371689 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\") pod \"78fb516f-09a1-4d73-a84d-6be7f78e360f\" (UID: \"78fb516f-09a1-4d73-a84d-6be7f78e360f\") " Oct 31 00:29:47.371962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.371908 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.372221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.372023 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.372467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.372447 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.372543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.372468 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.372543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.372478 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.373821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.373755 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:47.373927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.373861 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:47.373927 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.373866 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6" pod "78fb516f-09a1-4d73-a84d-6be7f78e360f" (UID: "78fb516f-09a1-4d73-a84d-6be7f78e360f"). InnerVolumeSpecName "kube-api-access-xc7w6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:47.399318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.399289 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb"] Oct 31 00:29:47.400022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400007 6234 cpu_manager.go:414] "containerMap: removing container" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400025 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400043 6234 cpu_manager.go:414] "containerMap: removing container" podUID="05f6f790-e53a-49c1-8b17-14c82caf5937" containerName="maas-api" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400048 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05f6f790-e53a-49c1-8b17-14c82caf5937" containerName="maas-api" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400054 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400059 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400069 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" containerName="payload-processing" Oct 31 00:29:47.400072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400073 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" containerName="payload-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400081 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400086 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400094 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6fa3b215-6d76-4d82-8f57-f23484ecf880" containerName="payload-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400098 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa3b215-6d76-4d82-8f57-f23484ecf880" containerName="payload-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400110 6234 cpu_manager.go:414] "containerMap: removing container" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400114 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400121 6234 cpu_manager.go:414] "containerMap: removing container" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400125 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400141 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400145 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400154 6234 cpu_manager.go:414] "containerMap: removing container" podUID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400158 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400166 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400170 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400180 6234 cpu_manager.go:414] "containerMap: removing container" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400184 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400191 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" containerName="payload-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400195 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" containerName="payload-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400204 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" containerName="payload-pre-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400209 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" containerName="payload-pre-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400217 6234 cpu_manager.go:414] "containerMap: removing container" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerName="payload-pre-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400221 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerName="payload-pre-processing" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400230 6234 cpu_manager.go:414] "containerMap: removing container" podUID="75097016-cdfc-4218-8c24-bee5d1bc31a1" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400234 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75097016-cdfc-4218-8c24-bee5d1bc31a1" containerName="maas-api" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400242 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400247 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400253 6234 cpu_manager.go:414] "containerMap: removing container" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400257 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400265 6234 cpu_manager.go:414] "containerMap: removing container" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400269 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400274 6234 cpu_manager.go:414] "containerMap: removing container" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400278 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400289 6234 cpu_manager.go:414] "containerMap: removing container" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" Oct 31 00:29:47.400284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400293 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400397 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400405 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400413 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="05f6f790-e53a-49c1-8b17-14c82caf5937" containerName="maas-api" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400419 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400425 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3486bbdd-f413-4abb-9f7a-50d56787a7f6" containerName="payload-processing" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400432 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400440 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa3b215-6d76-4d82-8f57-f23484ecf880" containerName="payload-processing" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400447 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="03c4be57-3cdc-49d9-8612-069310252339" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400452 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="bea6a00f-672b-4a94-971b-106c506294fd" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400460 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400465 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4abcb2ca-7779-4387-b0af-1579b448227b" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400470 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="88b26cc5-6eaf-4a46-8197-f63ffce6d8d1" containerName="maas-api" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400477 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d3aa20d4-7be0-48fb-9b28-c18b47f89b15" containerName="maas-api" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400482 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerName="main" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400488 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9546dbe9-0f87-4b6a-bc1a-3f5440c4b629" containerName="payload-processing" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400493 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9538ca46-c9b8-4fb6-9a75-ef8eb2e5a0e9" containerName="payload-pre-processing" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400499 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="50ff4ab8-21fa-415c-a775-f56c3fca9112" containerName="payload-pre-processing" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400505 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="75097016-cdfc-4218-8c24-bee5d1bc31a1" containerName="maas-api" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400511 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fa69024-0355-4478-afa2-ef91dffc4adf" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400518 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400522 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400527 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="db0d1e8b-f5dc-48a1-889e-f15d91e9eb88" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400533 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:47.401220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.400540 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f189999-8354-4304-8558-d88276d509cf" containerName="istio-proxy" Oct 31 00:29:47.404293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.404267 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.421476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.421445 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:47.421601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.421527 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:47.423962 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.423938 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb"] Oct 31 00:29:47.473859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.473762 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xc7w6\" (UniqueName: \"kubernetes.io/projected/78fb516f-09a1-4d73-a84d-6be7f78e360f-kube-api-access-xc7w6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.473859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.473811 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/78fb516f-09a1-4d73-a84d-6be7f78e360f-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.473859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.473826 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/78fb516f-09a1-4d73-a84d-6be7f78e360f-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:47.575210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.575167 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.575404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.575383 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wjnmt\" (UniqueName: \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.575489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.575434 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.676581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.676544 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wjnmt\" (UniqueName: \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.676774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.676609 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.676896 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.676844 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.677414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.677389 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.679503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.679477 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.689573 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.689550 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wjnmt\" (UniqueName: \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\") pod \"maas-api-revoke-sub-84bedf56-f89rb\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.723840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.723812 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:47.878869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.878133 6234 generic.go:356] "Generic (PLEG): container finished" podID="78fb516f-09a1-4d73-a84d-6be7f78e360f" containerID="9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765" exitCode=0 Oct 31 00:29:47.878869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.878272 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" event={"ID":"78fb516f-09a1-4d73-a84d-6be7f78e360f","Type":"ContainerDied","Data":"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765"} Oct 31 00:29:47.878869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.878298 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" event={"ID":"78fb516f-09a1-4d73-a84d-6be7f78e360f","Type":"ContainerDied","Data":"18d49fc0e10ce3b4fb06d494c60290b70963275c11a67a74a09e2223cbb4cded"} Oct 31 00:29:47.878869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.878314 6234 scope.go:122] "RemoveContainer" containerID="9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765" Oct 31 00:29:47.878869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.878419 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b" Oct 31 00:29:47.894464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.894388 6234 scope.go:122] "RemoveContainer" containerID="9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765" Oct 31 00:29:47.894717 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:47.894698 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765\": container with ID starting with 9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765 not found: ID does not exist" containerID="9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765" Oct 31 00:29:47.894769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.894728 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765"} err="failed to get container status \"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765\": rpc error: code = NotFound desc = could not find container \"9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765\": container with ID starting with 9bc352d35cb5f6ac501b84f867222eeb8b7fb49ee89949c341407439f14bc765 not found: ID does not exist" Oct 31 00:29:47.905746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.905718 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:29:47.912214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:47.912186 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-4c360e/distinct-w1-4c360e-kserve-747d7954ff-s6d6b"] Oct 31 00:29:48.112810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:48.063102 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb"] Oct 31 00:29:48.272824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:48.268888 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="78fb516f-09a1-4d73-a84d-6be7f78e360f" path="/var/lib/kubelet/pods/78fb516f-09a1-4d73-a84d-6be7f78e360f/volumes" Oct 31 00:29:48.891100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:48.890984 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" event={"ID":"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c","Type":"ContainerStarted","Data":"e6a8596e1129ccfd867c49f459c4b0ed0e58ebc6af1d99470363d14fd222eadd"} Oct 31 00:29:48.891100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:48.891060 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" event={"ID":"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c","Type":"ContainerStarted","Data":"2a25fdc6115bc6797051c209f3b1fbe6c8de2846db64edb34d6198ddef3e91bb"} Oct 31 00:29:48.915172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:48.915119 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" podStartSLOduration=1.915102903 podStartE2EDuration="1.915102903s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:48.91343561 +0000 UTC m=+2110.946531685" watchObservedRunningTime="2026-10-31 00:29:48.915102903 +0000 UTC m=+2110.948198972" Oct 31 00:29:49.929812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:49.929199 6234 generic.go:356] "Generic (PLEG): container finished" podID="e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" containerID="e6a8596e1129ccfd867c49f459c4b0ed0e58ebc6af1d99470363d14fd222eadd" exitCode=0 Oct 31 00:29:49.929812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:49.929318 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" event={"ID":"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c","Type":"ContainerDied","Data":"e6a8596e1129ccfd867c49f459c4b0ed0e58ebc6af1d99470363d14fd222eadd"} Oct 31 00:29:51.113003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.112980 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:51.181425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.181395 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\" (UniqueName: \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\") pod \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " Oct 31 00:29:51.181593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.181441 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\") pod \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " Oct 31 00:29:51.181681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.181619 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\") pod \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\" (UID: \"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c\") " Oct 31 00:29:51.181945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.181915 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca" pod "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" (UID: "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:51.182511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.182485 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-openshift-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.183968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.183939 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt" pod "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" (UID: "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c"). InnerVolumeSpecName "kube-api-access-wjnmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:51.184477 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.184448 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca" pod "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" (UID: "e59b72ac-f0b8-4a70-b5c9-7ce473faba1c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:51.283595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.283562 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-maas-api-xks-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.283595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.283591 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wjnmt\" (UniqueName: \"kubernetes.io/projected/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c-kube-api-access-wjnmt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:51.945665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.945631 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" Oct 31 00:29:51.945665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.945656 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb" event={"ID":"e59b72ac-f0b8-4a70-b5c9-7ce473faba1c","Type":"ContainerDied","Data":"2a25fdc6115bc6797051c209f3b1fbe6c8de2846db64edb34d6198ddef3e91bb"} Oct 31 00:29:51.945972 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:51.945734 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a25fdc6115bc6797051c209f3b1fbe6c8de2846db64edb34d6198ddef3e91bb" Oct 31 00:29:53.421884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.421850 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:29:53.422387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.422089 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" containerID="cri-o://0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084" gracePeriod=30 Oct 31 00:29:53.783439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.783401 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:29:53.783703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.783652 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" containerID="cri-o://504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13" gracePeriod=30 Oct 31 00:29:53.788676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.788649 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:29:53.788912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:53.788882 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" containerID="cri-o://f14f00fd1752376e60f36ab90b581aa390cdd27e4d604fc4eb80dbf33c645546" gracePeriod=30 Oct 31 00:29:54.552951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.552912 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z"] Oct 31 00:29:54.554995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.554970 6234 cpu_manager.go:414] "containerMap: removing container" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:54.555178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555162 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="277e0d55-a194-44cf-a75f-771a24a9fb65" containerName="istio-proxy" Oct 31 00:29:54.555316 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555305 6234 cpu_manager.go:414] "containerMap: removing container" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:54.555414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555402 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07b588a2-f55d-4fcc-bf3f-db6834789de6" containerName="istio-proxy" Oct 31 00:29:54.555507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555496 6234 cpu_manager.go:414] "containerMap: removing container" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:54.555578 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555567 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:54.555662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555653 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:54.555730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555720 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ac6ba56-d1b7-44fc-b8ce-4125405a30a9" containerName="istio-proxy" Oct 31 00:29:54.555838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555826 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" containerName="revoke-keys" Oct 31 00:29:54.555921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.555911 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" containerName="revoke-keys" Oct 31 00:29:54.556417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.556398 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="4229234f-54c4-40bd-804f-6ce0436a15a0" containerName="istio-proxy" Oct 31 00:29:54.556530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.556517 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" containerName="revoke-keys" Oct 31 00:29:54.561091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.561064 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.565032 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.565009 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z"] Oct 31 00:29:54.566909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.566890 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\"" Oct 31 00:29:54.568732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.568712 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:29:54.679082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.677882 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.680840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.680809 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmdpf\" (UniqueName: \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.682494 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.682470 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.784600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.784563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.784773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.784673 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmdpf\" (UniqueName: \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.784892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.784834 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.785484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.785465 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.787035 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.787012 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.799025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.799002 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmdpf\" (UniqueName: \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\") pod \"maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:54.879737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:54.879648 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:55.113712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:55.113659 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:29:55.113712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:55.113696 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:29:55.180029 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:55.179999 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z"] Oct 31 00:29:55.180942 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:29:55.180915 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8f437e9e_96fe_45ba_8757_c62b0e93319a.slice/crio-67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643 WatchSource:0}: Error finding container 67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643: Status 404 returned error can't find the container with id 67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643 Oct 31 00:29:56.044509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.043807 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" event={"ID":"8f437e9e-96fe-45ba-8757-c62b0e93319a","Type":"ContainerStarted","Data":"ef0081b8db8b57e5b263b268bef796e22bf27f565140ed232b5770901dd396c7"} Oct 31 00:29:56.044509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.043872 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" event={"ID":"8f437e9e-96fe-45ba-8757-c62b0e93319a","Type":"ContainerStarted","Data":"67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643"} Oct 31 00:29:56.065264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.065199 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" podStartSLOduration=2.065180685 podStartE2EDuration="2.065180685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:29:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:29:56.06282073 +0000 UTC m=+2118.095916801" watchObservedRunningTime="2026-10-31 00:29:56.065180685 +0000 UTC m=+2118.098276756" Oct 31 00:29:56.119449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.119414 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:56.119638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.119471 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:56.153614 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.153575 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:56.153754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.153633 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:56.397874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.397757 6234 patch_prober.go:29] interesting pod/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:29:56.397874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:56.397831 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:29:57.052351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:57.052316 6234 generic.go:356] "Generic (PLEG): container finished" podID="8f437e9e-96fe-45ba-8757-c62b0e93319a" containerID="ef0081b8db8b57e5b263b268bef796e22bf27f565140ed232b5770901dd396c7" exitCode=0 Oct 31 00:29:57.052763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:57.052369 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" event={"ID":"8f437e9e-96fe-45ba-8757-c62b0e93319a","Type":"ContainerDied","Data":"ef0081b8db8b57e5b263b268bef796e22bf27f565140ed232b5770901dd396c7"} Oct 31 00:29:58.296379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.296353 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:58.476536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.476500 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\") pod \"8f437e9e-96fe-45ba-8757-c62b0e93319a\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " Oct 31 00:29:58.476692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.476610 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\") pod \"8f437e9e-96fe-45ba-8757-c62b0e93319a\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " Oct 31 00:29:58.476753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.476708 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\" (UniqueName: \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\") pod \"8f437e9e-96fe-45ba-8757-c62b0e93319a\" (UID: \"8f437e9e-96fe-45ba-8757-c62b0e93319a\") " Oct 31 00:29:58.477000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.476971 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca" pod "8f437e9e-96fe-45ba-8757-c62b0e93319a" (UID: "8f437e9e-96fe-45ba-8757-c62b0e93319a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:58.477541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.477518 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-openshift-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.479012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.478983 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca" pod "8f437e9e-96fe-45ba-8757-c62b0e93319a" (UID: "8f437e9e-96fe-45ba-8757-c62b0e93319a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:29:58.479092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.479025 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf" pod "8f437e9e-96fe-45ba-8757-c62b0e93319a" (UID: "8f437e9e-96fe-45ba-8757-c62b0e93319a"). InnerVolumeSpecName "kube-api-access-hmdpf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:58.578984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.578950 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8f437e9e-96fe-45ba-8757-c62b0e93319a-maas-api-xks-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.578984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.578983 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmdpf\" (UniqueName: \"kubernetes.io/projected/8f437e9e-96fe-45ba-8757-c62b0e93319a-kube-api-access-hmdpf\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.686259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.686197 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:29:58.781162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781119 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781194 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781309 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781363 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781482 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781521 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781674 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781553 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.781813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.781652 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\") pod \"92427017-66b5-49af-8c7f-3c8867b057fe\" (UID: \"92427017-66b5-49af-8c7f-3c8867b057fe\") " Oct 31 00:29:58.782250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782077 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.782473 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782415 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.782802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782595 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.782802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782624 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.782802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782603 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:58.783008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.782920 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.783264 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.783232 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.784493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.784468 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:58.785007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.784987 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "kube-api-access-7s7h8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:58.785401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.785384 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:58.786857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.786836 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo" pod "92427017-66b5-49af-8c7f-3c8867b057fe" (UID: "92427017-66b5-49af-8c7f-3c8867b057fe"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:58.884181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884144 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s7h8\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-kube-api-access-7s7h8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884174 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92427017-66b5-49af-8c7f-3c8867b057fe-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884184 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884193 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884201 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92427017-66b5-49af-8c7f-3c8867b057fe-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884209 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92427017-66b5-49af-8c7f-3c8867b057fe-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:58.884404 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:58.884217 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92427017-66b5-49af-8c7f-3c8867b057fe-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.061591 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.061564 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:29:59.069524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.069494 6234 generic.go:356] "Generic (PLEG): container finished" podID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerID="f14f00fd1752376e60f36ab90b581aa390cdd27e4d604fc4eb80dbf33c645546" exitCode=0 Oct 31 00:29:59.069675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.069557 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" event={"ID":"0f45d136-993a-4d0f-a23b-0531bf9077bc","Type":"ContainerDied","Data":"f14f00fd1752376e60f36ab90b581aa390cdd27e4d604fc4eb80dbf33c645546"} Oct 31 00:29:59.071083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.071060 6234 generic.go:356] "Generic (PLEG): container finished" podID="956df451-22ed-4ae2-b363-df35b53bfc73" containerID="504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13" exitCode=0 Oct 31 00:29:59.071083 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.071101 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" Oct 31 00:29:59.071309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.071143 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" event={"ID":"956df451-22ed-4ae2-b363-df35b53bfc73","Type":"ContainerDied","Data":"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13"} Oct 31 00:29:59.071309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.071196 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv" event={"ID":"956df451-22ed-4ae2-b363-df35b53bfc73","Type":"ContainerDied","Data":"2e6d6cc15f73cbb46f8222aa388b0d3ec0f4e53721834646295e86a177e9648f"} Oct 31 00:29:59.071309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.071225 6234 scope.go:122] "RemoveContainer" containerID="504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13" Oct 31 00:29:59.073431 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.073409 6234 generic.go:356] "Generic (PLEG): container finished" podID="92427017-66b5-49af-8c7f-3c8867b057fe" containerID="0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084" exitCode=0 Oct 31 00:29:59.073545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.073468 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" Oct 31 00:29:59.073545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.073486 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerDied","Data":"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084"} Oct 31 00:29:59.073655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.073634 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h" event={"ID":"92427017-66b5-49af-8c7f-3c8867b057fe","Type":"ContainerDied","Data":"0ec75da9f95436b9ab8158660c3c8be3c21ed9bb920075d695b29a9f083d8202"} Oct 31 00:29:59.075037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.075022 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" Oct 31 00:29:59.075159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.075082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z" event={"ID":"8f437e9e-96fe-45ba-8757-c62b0e93319a","Type":"ContainerDied","Data":"67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643"} Oct 31 00:29:59.075159 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.075116 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="67459f8e69259a6586056e2c8232d7296365740dc587064912ab144ab4e03643" Oct 31 00:29:59.084699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.084678 6234 scope.go:122] "RemoveContainer" containerID="504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13" Oct 31 00:29:59.085024 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:59.084997 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13\": container with ID starting with 504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13 not found: ID does not exist" containerID="504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13" Oct 31 00:29:59.085103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.085035 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13"} err="failed to get container status \"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13\": rpc error: code = NotFound desc = could not find container \"504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13\": container with ID starting with 504b714b770e3d50d985926f66595ce552ab2aecf85859e477016a9aec207d13 not found: ID does not exist" Oct 31 00:29:59.085103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.085059 6234 scope.go:122] "RemoveContainer" containerID="0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084" Oct 31 00:29:59.085891 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.085867 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.085912 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.085961 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086022 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086068 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086111 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086119 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086138 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086241 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086333 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\") pod \"956df451-22ed-4ae2-b363-df35b53bfc73\" (UID: \"956df451-22ed-4ae2-b363-df35b53bfc73\") " Oct 31 00:29:59.086587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086353 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.086715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086663 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.086978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086960 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.087049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.086990 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.087166 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.087149 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.087233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.087174 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.087233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.087189 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.087233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.087203 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.087233 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.087217 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/956df451-22ed-4ae2-b363-df35b53bfc73-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.089137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.089112 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.089485 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.089452 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.097487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.090634 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.097487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.092076 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4" pod "956df451-22ed-4ae2-b363-df35b53bfc73" (UID: "956df451-22ed-4ae2-b363-df35b53bfc73"). InnerVolumeSpecName "kube-api-access-vr5d4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.099906 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.099884 6234 scope.go:122] "RemoveContainer" containerID="2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624" Oct 31 00:29:59.112678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.112648 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:29:59.117530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.117509 6234 scope.go:122] "RemoveContainer" containerID="0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084" Oct 31 00:29:59.117947 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:59.117821 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084\": container with ID starting with 0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084 not found: ID does not exist" containerID="0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084" Oct 31 00:29:59.117947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.117867 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084"} err="failed to get container status \"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084\": rpc error: code = NotFound desc = could not find container \"0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084\": container with ID starting with 0ce4ff7c159b07982f806e14b356c1b1a8f146aeb49ab1eb0dc4fef165c46084 not found: ID does not exist" Oct 31 00:29:59.117947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.117892 6234 scope.go:122] "RemoveContainer" containerID="2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624" Oct 31 00:29:59.119147 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:29:59.119121 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624\": container with ID starting with 2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624 not found: ID does not exist" containerID="2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624" Oct 31 00:29:59.119241 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.119157 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624"} err="failed to get container status \"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624\": rpc error: code = NotFound desc = could not find container \"2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624\": container with ID starting with 2da424a59b6488448c8a965348be06486e80177b2a3effd8629476003fe02624 not found: ID does not exist" Oct 31 00:29:59.119305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.119244 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-cc0fe1-openshift-default-7876b5db45-2472h"] Oct 31 00:29:59.121675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.121655 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:29:59.235862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.225386 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/956df451-22ed-4ae2-b363-df35b53bfc73-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.235862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.225453 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/956df451-22ed-4ae2-b363-df35b53bfc73-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.235862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.225476 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.235862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.225490 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vr5d4\" (UniqueName: \"kubernetes.io/projected/956df451-22ed-4ae2-b363-df35b53bfc73-kube-api-access-vr5d4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.311206 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.311171 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:29:59.326334 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326300 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326364 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326424 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326505 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326590 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326628 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326704 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.326852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326719 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326876 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.326965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326854 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:29:59.326965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326940 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.327129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.326984 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\") pod \"0f45d136-993a-4d0f-a23b-0531bf9077bc\" (UID: \"0f45d136-993a-4d0f-a23b-0531bf9077bc\") " Oct 31 00:29:59.327197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327172 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.327253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327193 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.327593 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327574 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.327743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327598 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.327743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327614 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.327743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327629 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.327743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.327642 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0f45d136-993a-4d0f-a23b-0531bf9077bc-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.329503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.329476 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:29:59.329503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.329489 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:29:59.330004 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.329979 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "kube-api-access-725qg". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.333002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.332958 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token" pod "0f45d136-993a-4d0f-a23b-0531bf9077bc" (UID: "0f45d136-993a-4d0f-a23b-0531bf9077bc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:29:59.415105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.415064 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:29:59.429538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.429496 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.429527 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.429537 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0f45d136-993a-4d0f-a23b-0531bf9077bc-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.429538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.429548 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-725qg\" (UniqueName: \"kubernetes.io/projected/0f45d136-993a-4d0f-a23b-0531bf9077bc-kube-api-access-725qg\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:29:59.430451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:29:59.430425 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-kbzcv"] Oct 31 00:30:00.083746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.083703 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" path="/var/lib/kubelet/pods/92427017-66b5-49af-8c7f-3c8867b057fe/volumes" Oct 31 00:30:00.084355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.084322 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" path="/var/lib/kubelet/pods/956df451-22ed-4ae2-b363-df35b53bfc73/volumes" Oct 31 00:30:00.085210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.085191 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" Oct 31 00:30:00.085332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.085225 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4" event={"ID":"0f45d136-993a-4d0f-a23b-0531bf9077bc","Type":"ContainerDied","Data":"9d20938719159fc8e03de26a75e93416d9e1d77c1d17c2bc10ec704cfc45edf9"} Oct 31 00:30:00.085332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.085294 6234 scope.go:122] "RemoveContainer" containerID="f14f00fd1752376e60f36ab90b581aa390cdd27e4d604fc4eb80dbf33c645546" Oct 31 00:30:00.122314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.122279 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:30:00.139197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.139161 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w4-a39be4-openshift-default-78d588db45-pjfg4"] Oct 31 00:30:00.189992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.189958 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:00.190646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190631 6234 cpu_manager.go:414] "containerMap: removing container" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.190690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190649 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.190690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190663 6234 cpu_manager.go:414] "containerMap: removing container" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.190690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190668 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190705 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8f437e9e-96fe-45ba-8757-c62b0e93319a" containerName="revoke-keys" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190709 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8f437e9e-96fe-45ba-8757-c62b0e93319a" containerName="revoke-keys" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190718 6234 cpu_manager.go:414] "containerMap: removing container" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190723 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190739 6234 cpu_manager.go:414] "containerMap: removing container" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" Oct 31 00:30:00.190810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190744 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" Oct 31 00:30:00.191002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190842 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8f437e9e-96fe-45ba-8757-c62b0e93319a" containerName="revoke-keys" Oct 31 00:30:00.191002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190851 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="956df451-22ed-4ae2-b363-df35b53bfc73" containerName="istio-proxy" Oct 31 00:30:00.191002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190861 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" containerName="istio-proxy" Oct 31 00:30:00.191002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190869 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.191002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.190874 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="92427017-66b5-49af-8c7f-3c8867b057fe" containerName="istio-proxy" Oct 31 00:30:00.196182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.196157 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.215686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.215652 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:00.247551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.247519 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:00.248892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.248863 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z9l7s\" (UniqueName: \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.249040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.249011 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.253822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.253774 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.262194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.262164 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:00.273011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.272986 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.273181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.273010 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:00.294329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.294295 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:00.303026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.302987 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-nrj88\"" Oct 31 00:30:00.316601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.316569 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:00.322028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.322003 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.354718 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jprk8\" (UniqueName: \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.354775 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbg7k\" (UniqueName: \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.354839 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.354914 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z9l7s\" (UniqueName: \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.355011 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jvnss\" (UniqueName: \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.355059 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.355207 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.361240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.355261 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.362544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.362507 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.389810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.378849 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:00.436515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.436446 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z9l7s\" (UniqueName: \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\") pod \"maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.456872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.456834 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.457072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.456916 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jprk8\" (UniqueName: \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.457072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.456934 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbg7k\" (UniqueName: \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.457072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.456958 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.457072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.457069 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jvnss\" (UniqueName: \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.457306 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.457210 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.457623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.457599 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.457915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.457896 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.457985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.457895 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.475598 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.475569 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jprk8\" (UniqueName: \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.493219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.493183 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jvnss\" (UniqueName: \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\") pod \"maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.502537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.502498 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbg7k\" (UniqueName: \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\") pod \"maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.513539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.513502 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:00.579192 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.579161 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:00.594039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.593436 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:00.690476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.690188 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:00.818878 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:30:00.818832 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6773cff0_24af_4eaf_a2e3_7cab7531ef7a.slice/crio-e130aae0bfb58efd605e0e343160518c552f10c324413bc1ec5c243c0c1abf8a WatchSource:0}: Error finding container e130aae0bfb58efd605e0e343160518c552f10c324413bc1ec5c243c0c1abf8a: Status 404 returned error can't find the container with id e130aae0bfb58efd605e0e343160518c552f10c324413bc1ec5c243c0c1abf8a Oct 31 00:30:00.820709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.820681 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:00.894986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.894958 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:00.896625 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:30:00.896602 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c514e69_2fcc_4c62_a390_eddc0863076d.slice/crio-4d7413ff0bd44e02954194e3f7bd79a82198bf06e7b69238b3321a0f528b1fd5 WatchSource:0}: Error finding container 4d7413ff0bd44e02954194e3f7bd79a82198bf06e7b69238b3321a0f528b1fd5: Status 404 returned error can't find the container with id 4d7413ff0bd44e02954194e3f7bd79a82198bf06e7b69238b3321a0f528b1fd5 Oct 31 00:30:00.916084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.916047 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:00.917003 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:30:00.916971 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb447616e_0dc2_4299_b91a_e4b22ee30867.slice/crio-351be2dff179808b2429836cfaf6afa5b6a6fbc51e689e63571bc314b412a020 WatchSource:0}: Error finding container 351be2dff179808b2429836cfaf6afa5b6a6fbc51e689e63571bc314b412a020: Status 404 returned error can't find the container with id 351be2dff179808b2429836cfaf6afa5b6a6fbc51e689e63571bc314b412a020 Oct 31 00:30:00.997168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:00.997139 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:01.043205 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:30:01.043174 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda0eacb9_63e0_4721_b3a1_d91e386cccbe.slice/crio-871f2c5ccfea1cdfd508f48e0ca3e0197fc87fe8a5945b43dfde14d78c8facef WatchSource:0}: Error finding container 871f2c5ccfea1cdfd508f48e0ca3e0197fc87fe8a5945b43dfde14d78c8facef: Status 404 returned error can't find the container with id 871f2c5ccfea1cdfd508f48e0ca3e0197fc87fe8a5945b43dfde14d78c8facef Oct 31 00:30:01.097191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:01.097083 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerStarted","Data":"4d7413ff0bd44e02954194e3f7bd79a82198bf06e7b69238b3321a0f528b1fd5"} Oct 31 00:30:01.098558 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:01.098532 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerStarted","Data":"871f2c5ccfea1cdfd508f48e0ca3e0197fc87fe8a5945b43dfde14d78c8facef"} Oct 31 00:30:01.100731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:01.100707 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerStarted","Data":"e130aae0bfb58efd605e0e343160518c552f10c324413bc1ec5c243c0c1abf8a"} Oct 31 00:30:01.102016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:01.101989 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerStarted","Data":"351be2dff179808b2429836cfaf6afa5b6a6fbc51e689e63571bc314b412a020"} Oct 31 00:30:02.083063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.083028 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f45d136-993a-4d0f-a23b-0531bf9077bc" path="/var/lib/kubelet/pods/0f45d136-993a-4d0f-a23b-0531bf9077bc/volumes" Oct 31 00:30:02.113080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.113030 6234 generic.go:356] "Generic (PLEG): container finished" podID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerID="d767b110f3d01a6b28d227400a5118572788cfeaf2a3f2fd78334399c1dacb77" exitCode=60 Oct 31 00:30:02.113289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.113116 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerDied","Data":"d767b110f3d01a6b28d227400a5118572788cfeaf2a3f2fd78334399c1dacb77"} Oct 31 00:30:02.113565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.113543 6234 scope.go:122] "RemoveContainer" containerID="d767b110f3d01a6b28d227400a5118572788cfeaf2a3f2fd78334399c1dacb77" Oct 31 00:30:02.115621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.115579 6234 generic.go:356] "Generic (PLEG): container finished" podID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerID="9efe3168d230ab49d63b19b215f8bc65d44e173355674fd32af9fa690cfd2213" exitCode=60 Oct 31 00:30:02.115719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.115695 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerDied","Data":"9efe3168d230ab49d63b19b215f8bc65d44e173355674fd32af9fa690cfd2213"} Oct 31 00:30:02.115977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.115957 6234 scope.go:122] "RemoveContainer" containerID="9efe3168d230ab49d63b19b215f8bc65d44e173355674fd32af9fa690cfd2213" Oct 31 00:30:02.117641 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.117612 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerStarted","Data":"83ffc4c45b0c80335e1a7a1c45463ea9e14fc82da616bd0d599b3cc74b562ad8"} Oct 31 00:30:02.119489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.119462 6234 generic.go:356] "Generic (PLEG): container finished" podID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerID="9f3030d7151bba8b4e37ca8d28eff59b479723cc6f490e39b267f6fb8d781319" exitCode=60 Oct 31 00:30:02.119575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.119514 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerDied","Data":"9f3030d7151bba8b4e37ca8d28eff59b479723cc6f490e39b267f6fb8d781319"} Oct 31 00:30:02.119830 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.119814 6234 scope.go:122] "RemoveContainer" containerID="9f3030d7151bba8b4e37ca8d28eff59b479723cc6f490e39b267f6fb8d781319" Oct 31 00:30:02.186818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.186728 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" podStartSLOduration=2.186711659 podStartE2EDuration="2.186711659s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:02.185333773 +0000 UTC m=+2124.218429844" watchObservedRunningTime="2026-10-31 00:30:02.186711659 +0000 UTC m=+2124.219807730" Oct 31 00:30:02.512568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.512539 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:30:02.515966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.515930 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:30:02.589760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.589668 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\") pod \"ad4bbafa-37e8-403d-84a6-6c4437086688\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " Oct 31 00:30:02.589970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.589806 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\" (UniqueName: \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\") pod \"ad4bbafa-37e8-403d-84a6-6c4437086688\" (UID: \"ad4bbafa-37e8-403d-84a6-6c4437086688\") " Oct 31 00:30:02.590020 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.589979 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\") pod \"f1f38638-649c-42ae-afac-6eb04bfbdd91\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " Oct 31 00:30:02.590144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.590125 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\" (UniqueName: \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\") pod \"f1f38638-649c-42ae-afac-6eb04bfbdd91\" (UID: \"f1f38638-649c-42ae-afac-6eb04bfbdd91\") " Oct 31 00:30:02.590253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.590155 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume" pod "ad4bbafa-37e8-403d-84a6-6c4437086688" (UID: "ad4bbafa-37e8-403d-84a6-6c4437086688"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:02.590407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.590384 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume" pod "f1f38638-649c-42ae-afac-6eb04bfbdd91" (UID: "f1f38638-649c-42ae-afac-6eb04bfbdd91"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:02.590721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.590701 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f1f38638-649c-42ae-afac-6eb04bfbdd91-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:02.590828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.590728 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ad4bbafa-37e8-403d-84a6-6c4437086688-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:02.592731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.592710 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764" pod "f1f38638-649c-42ae-afac-6eb04bfbdd91" (UID: "f1f38638-649c-42ae-afac-6eb04bfbdd91"). InnerVolumeSpecName "kube-api-access-ld764". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:02.593211 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.593188 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq" pod "ad4bbafa-37e8-403d-84a6-6c4437086688" (UID: "ad4bbafa-37e8-403d-84a6-6c4437086688"). InnerVolumeSpecName "kube-api-access-98jjq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:02.697856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.697718 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ld764\" (UniqueName: \"kubernetes.io/projected/f1f38638-649c-42ae-afac-6eb04bfbdd91-kube-api-access-ld764\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:02.697856 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:02.697756 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-98jjq\" (UniqueName: \"kubernetes.io/projected/ad4bbafa-37e8-403d-84a6-6c4437086688-kube-api-access-98jjq\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:03.129868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.129818 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerStarted","Data":"084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c"} Oct 31 00:30:03.131313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.131288 6234 generic.go:356] "Generic (PLEG): container finished" podID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerID="9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c" exitCode=0 Oct 31 00:30:03.131440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.131332 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" Oct 31 00:30:03.131440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.131363 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" event={"ID":"ad4bbafa-37e8-403d-84a6-6c4437086688","Type":"ContainerDied","Data":"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c"} Oct 31 00:30:03.131440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.131399 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk" event={"ID":"ad4bbafa-37e8-403d-84a6-6c4437086688","Type":"ContainerDied","Data":"271f393f8037b8cf38fb5b6fd41f3f651cf294e02d0ca08c3cd2dc203513403e"} Oct 31 00:30:03.131440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.131419 6234 scope.go:122] "RemoveContainer" containerID="9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c" Oct 31 00:30:03.133008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.132982 6234 generic.go:356] "Generic (PLEG): container finished" podID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerID="df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d" exitCode=0 Oct 31 00:30:03.133104 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.133027 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" Oct 31 00:30:03.133104 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.133036 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" event={"ID":"f1f38638-649c-42ae-afac-6eb04bfbdd91","Type":"ContainerDied","Data":"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d"} Oct 31 00:30:03.133207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.133082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt" event={"ID":"f1f38638-649c-42ae-afac-6eb04bfbdd91","Type":"ContainerDied","Data":"8a890a49981472787b1fb4e9d793b82dec0b89f2651d58a54648e193e55358e1"} Oct 31 00:30:03.138610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.138575 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerStarted","Data":"d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b"} Oct 31 00:30:03.140694 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.140664 6234 generic.go:356] "Generic (PLEG): container finished" podID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerID="83ffc4c45b0c80335e1a7a1c45463ea9e14fc82da616bd0d599b3cc74b562ad8" exitCode=60 Oct 31 00:30:03.140866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.140846 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerDied","Data":"83ffc4c45b0c80335e1a7a1c45463ea9e14fc82da616bd0d599b3cc74b562ad8"} Oct 31 00:30:03.141223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.141196 6234 scope.go:122] "RemoveContainer" containerID="83ffc4c45b0c80335e1a7a1c45463ea9e14fc82da616bd0d599b3cc74b562ad8" Oct 31 00:30:03.145661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.145631 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerStarted","Data":"4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61"} Oct 31 00:30:03.146372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.146351 6234 scope.go:122] "RemoveContainer" containerID="9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c" Oct 31 00:30:03.146660 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:03.146638 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c\": container with ID starting with 9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c not found: ID does not exist" containerID="9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c" Oct 31 00:30:03.146764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.146670 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c"} err="failed to get container status \"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c\": rpc error: code = NotFound desc = could not find container \"9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c\": container with ID starting with 9b4adbac246c2e862023e00170c5a6fcdd4178e6be734d59964e812587cd117c not found: ID does not exist" Oct 31 00:30:03.146764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.146689 6234 scope.go:122] "RemoveContainer" containerID="df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d" Oct 31 00:30:03.156219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.156142 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" podStartSLOduration=3.15612515 podStartE2EDuration="3.15612515s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.151291738 +0000 UTC m=+2125.184387906" watchObservedRunningTime="2026-10-31 00:30:03.15612515 +0000 UTC m=+2125.189221221" Oct 31 00:30:03.172238 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:03.172204 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:30:03.186207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.186175 6234 scope.go:122] "RemoveContainer" containerID="df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d" Oct 31 00:30:03.186597 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:03.186576 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d\": container with ID starting with df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d not found: ID does not exist" containerID="df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d" Oct 31 00:30:03.186646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.186609 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d"} err="failed to get container status \"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d\": rpc error: code = NotFound desc = could not find container \"df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d\": container with ID starting with df93527d6ad0e3776b8c2c6225272b3686d730bb92793849bbdd41917c717e6d not found: ID does not exist" Oct 31 00:30:03.242391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.242337 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" podStartSLOduration=3.242323336 podStartE2EDuration="3.242323336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.241726898 +0000 UTC m=+2125.274822969" watchObservedRunningTime="2026-10-31 00:30:03.242323336 +0000 UTC m=+2125.275419406" Oct 31 00:30:03.272118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.272061 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" podStartSLOduration=3.272046138 podStartE2EDuration="3.272046138s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:30:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:30:03.269427672 +0000 UTC m=+2125.302523757" watchObservedRunningTime="2026-10-31 00:30:03.272046138 +0000 UTC m=+2125.305142234" Oct 31 00:30:03.291766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.291731 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:30:03.298456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.298411 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w1-4c360e-75c5967dc9-vp2mt"] Oct 31 00:30:03.313072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.313036 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:30:03.317979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:03.317933 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w1-4c360e-84657dd9f-7v2vk"] Oct 31 00:30:04.082574 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.082495 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" path="/var/lib/kubelet/pods/ad4bbafa-37e8-403d-84a6-6c4437086688/volumes" Oct 31 00:30:04.083040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.083024 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" path="/var/lib/kubelet/pods/f1f38638-649c-42ae-afac-6eb04bfbdd91/volumes" Oct 31 00:30:04.157742 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.157712 6234 generic.go:356] "Generic (PLEG): container finished" podID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerID="d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b" exitCode=60 Oct 31 00:30:04.158209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.157801 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerDied","Data":"d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b"} Oct 31 00:30:04.158209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.157867 6234 scope.go:122] "RemoveContainer" containerID="9efe3168d230ab49d63b19b215f8bc65d44e173355674fd32af9fa690cfd2213" Oct 31 00:30:04.158209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.158187 6234 scope.go:122] "RemoveContainer" containerID="d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b" Oct 31 00:30:04.158584 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:04.158558 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr_odh-ai-gateway-infra(9c514e69-2fcc-4c62-a390-eddc0863076d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" Oct 31 00:30:04.159982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.159950 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerStarted","Data":"ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc"} Oct 31 00:30:04.160227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.160214 6234 scope.go:122] "RemoveContainer" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" Oct 31 00:30:04.160570 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:04.160543 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx_odh-ai-gateway-infra(da0eacb9-63e0-4721-b3a1-d91e386cccbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" Oct 31 00:30:04.161848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.161820 6234 generic.go:356] "Generic (PLEG): container finished" podID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerID="4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61" exitCode=60 Oct 31 00:30:04.161917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.161904 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerDied","Data":"4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61"} Oct 31 00:30:04.162133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.162118 6234 scope.go:122] "RemoveContainer" containerID="4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61" Oct 31 00:30:04.162442 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:04.162420 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv_odh-ai-gateway-infra(6773cff0-24af-4eaf-a2e3-7cab7531ef7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" Oct 31 00:30:04.163912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.163862 6234 generic.go:356] "Generic (PLEG): container finished" podID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerID="084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c" exitCode=60 Oct 31 00:30:04.164002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.163928 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerDied","Data":"084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c"} Oct 31 00:30:04.164182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.164168 6234 scope.go:122] "RemoveContainer" containerID="084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c" Oct 31 00:30:04.164449 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:04.164432 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp_odh-ai-gateway-infra(b447616e-0dc2-4299-b91a-e4b22ee30867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" Oct 31 00:30:04.171505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.171483 6234 scope.go:122] "RemoveContainer" containerID="9f3030d7151bba8b4e37ca8d28eff59b479723cc6f490e39b267f6fb8d781319" Oct 31 00:30:04.186645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.186594 6234 scope.go:122] "RemoveContainer" containerID="d767b110f3d01a6b28d227400a5118572788cfeaf2a3f2fd78334399c1dacb77" Oct 31 00:30:04.855809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:04.851988 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:30:05.183123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.183031 6234 scope.go:122] "RemoveContainer" containerID="d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b" Oct 31 00:30:05.183556 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:05.183453 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr_odh-ai-gateway-infra(9c514e69-2fcc-4c62-a390-eddc0863076d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" Oct 31 00:30:05.185434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.185405 6234 generic.go:356] "Generic (PLEG): container finished" podID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" exitCode=60 Oct 31 00:30:05.187318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.185532 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerDied","Data":"ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc"} Oct 31 00:30:05.187318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.185631 6234 scope.go:122] "RemoveContainer" containerID="83ffc4c45b0c80335e1a7a1c45463ea9e14fc82da616bd0d599b3cc74b562ad8" Oct 31 00:30:05.188738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.188718 6234 scope.go:122] "RemoveContainer" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" Oct 31 00:30:05.189290 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:05.189262 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx_odh-ai-gateway-infra(da0eacb9-63e0-4721-b3a1-d91e386cccbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" Oct 31 00:30:05.191899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.191874 6234 scope.go:122] "RemoveContainer" containerID="4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61" Oct 31 00:30:05.192324 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:05.192296 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv_odh-ai-gateway-infra(6773cff0-24af-4eaf-a2e3-7cab7531ef7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" Oct 31 00:30:05.195291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:05.195265 6234 scope.go:122] "RemoveContainer" containerID="084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c" Oct 31 00:30:05.195677 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:05.195648 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp_odh-ai-gateway-infra(b447616e-0dc2-4299-b91a-e4b22ee30867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" Oct 31 00:30:06.204077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:06.204051 6234 scope.go:122] "RemoveContainer" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" Oct 31 00:30:06.204455 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:06.204383 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx_odh-ai-gateway-infra(da0eacb9-63e0-4721-b3a1-d91e386cccbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" Oct 31 00:30:09.852817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:09.852750 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" containerID="cri-o://ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24" gracePeriod=25 Oct 31 00:30:12.168031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:12.167988 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:30:14.153718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.153688 6234 scope.go:122] "RemoveContainer" containerID="d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b" Oct 31 00:30:14.335406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.335346 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" containerID="cri-o://ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847" gracePeriod=45 Oct 31 00:30:14.621815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.621770 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:30:14.733905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.733825 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734029 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734053 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.734101 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734071 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\" (UniqueName: \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734192 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734177 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734229 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\" (UniqueName: \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734277 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\") pod \"0d148c2b-1b9f-4782-acbe-42c5b821a395\" (UID: \"0d148c2b-1b9f-4782-acbe-42c5b821a395\") " Oct 31 00:30:14.734355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.734432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734410 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.734930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734853 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.734930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734879 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.734930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.734893 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.736426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.736396 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:14.736426 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.736402 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:14.736563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.736484 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr" pod "0d148c2b-1b9f-4782-acbe-42c5b821a395" (UID: "0d148c2b-1b9f-4782-acbe-42c5b821a395"). InnerVolumeSpecName "kube-api-access-lnxgr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:14.779036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.779001 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:30:14.835989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.835950 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:30:14.837151 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.837119 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0d148c2b-1b9f-4782-acbe-42c5b821a395-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.837301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.837157 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lnxgr\" (UniqueName: \"kubernetes.io/projected/0d148c2b-1b9f-4782-acbe-42c5b821a395-kube-api-access-lnxgr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:14.837301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:14.837175 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0d148c2b-1b9f-4782-acbe-42c5b821a395-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:15.075268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.075162 6234 scope.go:122] "RemoveContainer" containerID="4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61" Oct 31 00:30:15.075445 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.075341 6234 scope.go:122] "RemoveContainer" containerID="084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c" Oct 31 00:30:15.348917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.348885 6234 generic.go:356] "Generic (PLEG): container finished" podID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerID="0dbfaca51b7ad8a686a525ef0b2aac304b71ee7633669eaa1686e558e40362db" exitCode=60 Oct 31 00:30:15.349315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.348953 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerDied","Data":"0dbfaca51b7ad8a686a525ef0b2aac304b71ee7633669eaa1686e558e40362db"} Oct 31 00:30:15.349315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.349016 6234 scope.go:122] "RemoveContainer" containerID="d649b15594cda0c16a06ed213737cf55ec887b6464d87a84b5e578dc20d53f7b" Oct 31 00:30:15.349504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.349487 6234 scope.go:122] "RemoveContainer" containerID="0dbfaca51b7ad8a686a525ef0b2aac304b71ee7633669eaa1686e558e40362db" Oct 31 00:30:15.349910 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:15.349885 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr_odh-ai-gateway-infra(9c514e69-2fcc-4c62-a390-eddc0863076d)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" Oct 31 00:30:15.350563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.350543 6234 generic.go:356] "Generic (PLEG): container finished" podID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerID="ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847" exitCode=0 Oct 31 00:30:15.350638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.350585 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" event={"ID":"0d148c2b-1b9f-4782-acbe-42c5b821a395","Type":"ContainerDied","Data":"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847"} Oct 31 00:30:15.350638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.350626 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" event={"ID":"0d148c2b-1b9f-4782-acbe-42c5b821a395","Type":"ContainerDied","Data":"d1ab59922f91b9304f836610c820612afdd3040921f305a2a10d89a7a97bbb79"} Oct 31 00:30:15.350741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.350596 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2" Oct 31 00:30:15.364452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.364435 6234 scope.go:122] "RemoveContainer" containerID="ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847" Oct 31 00:30:15.374513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.374491 6234 scope.go:122] "RemoveContainer" containerID="ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847" Oct 31 00:30:15.374849 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:15.374825 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847\": container with ID starting with ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847 not found: ID does not exist" containerID="ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847" Oct 31 00:30:15.374919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.374860 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847"} err="failed to get container status \"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847\": rpc error: code = NotFound desc = could not find container \"ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847\": container with ID starting with ea558b2b1d01519b288d777444f23dba301d4f81d38ca58bc470a6d08b55c847 not found: ID does not exist" Oct 31 00:30:15.398031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.397992 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:30:15.402812 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:15.402763 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-d91de6/distinct-w4-d91de6-kserve-545d4d87b-sbzw2"] Oct 31 00:30:16.091744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.091705 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" path="/var/lib/kubelet/pods/0d148c2b-1b9f-4782-acbe-42c5b821a395/volumes" Oct 31 00:30:16.359288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.359198 6234 generic.go:356] "Generic (PLEG): container finished" podID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerID="4320522dcf4ffe3063c80daf296fde07fee79c90106a84011a4b3c453368ef01" exitCode=60 Oct 31 00:30:16.359288 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.359267 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerDied","Data":"4320522dcf4ffe3063c80daf296fde07fee79c90106a84011a4b3c453368ef01"} Oct 31 00:30:16.359825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.359324 6234 scope.go:122] "RemoveContainer" containerID="4bba1f962f031353a6e6910b865951016ded9e42e83cb3c09d70b55a26ce8c61" Oct 31 00:30:16.359883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.359838 6234 scope.go:122] "RemoveContainer" containerID="4320522dcf4ffe3063c80daf296fde07fee79c90106a84011a4b3c453368ef01" Oct 31 00:30:16.360541 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:16.360219 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv_odh-ai-gateway-infra(6773cff0-24af-4eaf-a2e3-7cab7531ef7a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" Oct 31 00:30:16.362138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.362119 6234 generic.go:356] "Generic (PLEG): container finished" podID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerID="ed1c874b0361df2805b6f8f124f767cd5b5b3b489b061b6130f884244da4a30d" exitCode=60 Oct 31 00:30:16.362226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.362187 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerDied","Data":"ed1c874b0361df2805b6f8f124f767cd5b5b3b489b061b6130f884244da4a30d"} Oct 31 00:30:16.362574 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.362554 6234 scope.go:122] "RemoveContainer" containerID="ed1c874b0361df2805b6f8f124f767cd5b5b3b489b061b6130f884244da4a30d" Oct 31 00:30:16.362906 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:16.362886 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp_odh-ai-gateway-infra(b447616e-0dc2-4299-b91a-e4b22ee30867)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" Oct 31 00:30:16.372589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.372568 6234 scope.go:122] "RemoveContainer" containerID="084510f7aaa5d1466b2e116f0a238c3b56f6cc184afda1cdd711dc4ea3cdd41c" Oct 31 00:30:16.425837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.425735 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:16.592622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.592592 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:16.677696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.670395 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\") pod \"9c514e69-2fcc-4c62-a390-eddc0863076d\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " Oct 31 00:30:16.677696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.670623 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\" (UniqueName: \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\") pod \"9c514e69-2fcc-4c62-a390-eddc0863076d\" (UID: \"9c514e69-2fcc-4c62-a390-eddc0863076d\") " Oct 31 00:30:16.677696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.671204 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca" pod "9c514e69-2fcc-4c62-a390-eddc0863076d" (UID: "9c514e69-2fcc-4c62-a390-eddc0863076d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:16.677696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.671300 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9c514e69-2fcc-4c62-a390-eddc0863076d-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:16.677696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.673631 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8" pod "9c514e69-2fcc-4c62-a390-eddc0863076d" (UID: "9c514e69-2fcc-4c62-a390-eddc0863076d"). InnerVolumeSpecName "kube-api-access-jprk8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:16.772397 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:16.772350 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jprk8\" (UniqueName: \"kubernetes.io/projected/9c514e69-2fcc-4c62-a390-eddc0863076d-kube-api-access-jprk8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.376761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.376728 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" Oct 31 00:30:17.376761 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.376741 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr" event={"ID":"9c514e69-2fcc-4c62-a390-eddc0863076d","Type":"ContainerDied","Data":"4d7413ff0bd44e02954194e3f7bd79a82198bf06e7b69238b3321a0f528b1fd5"} Oct 31 00:30:17.377315 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.376835 6234 scope.go:122] "RemoveContainer" containerID="0dbfaca51b7ad8a686a525ef0b2aac304b71ee7633669eaa1686e558e40362db" Oct 31 00:30:17.425802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.425750 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:17.441754 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.441717 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:17.455847 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.455813 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:17.459011 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.458987 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-a8d8bf-29890110-64wfr"] Oct 31 00:30:17.698335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.698297 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:17.768382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.768355 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:17.800258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800221 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\") pod \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " Oct 31 00:30:17.800433 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800317 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\" (UniqueName: \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\") pod \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\" (UID: \"6773cff0-24af-4eaf-a2e3-7cab7531ef7a\") " Oct 31 00:30:17.800433 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800338 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\" (UniqueName: \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\") pod \"b447616e-0dc2-4299-b91a-e4b22ee30867\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " Oct 31 00:30:17.800600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800576 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\") pod \"b447616e-0dc2-4299-b91a-e4b22ee30867\" (UID: \"b447616e-0dc2-4299-b91a-e4b22ee30867\") " Oct 31 00:30:17.800717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800562 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca" pod "6773cff0-24af-4eaf-a2e3-7cab7531ef7a" (UID: "6773cff0-24af-4eaf-a2e3-7cab7531ef7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.800935 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.800910 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca" pod "b447616e-0dc2-4299-b91a-e4b22ee30867" (UID: "b447616e-0dc2-4299-b91a-e4b22ee30867"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:17.801197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.801144 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.801197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.801162 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b447616e-0dc2-4299-b91a-e4b22ee30867-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.802687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.802668 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s" pod "6773cff0-24af-4eaf-a2e3-7cab7531ef7a" (UID: "6773cff0-24af-4eaf-a2e3-7cab7531ef7a"). InnerVolumeSpecName "kube-api-access-z9l7s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.802772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.802695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss" pod "b447616e-0dc2-4299-b91a-e4b22ee30867" (UID: "b447616e-0dc2-4299-b91a-e4b22ee30867"). InnerVolumeSpecName "kube-api-access-jvnss". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:17.903020 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.902937 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z9l7s\" (UniqueName: \"kubernetes.io/projected/6773cff0-24af-4eaf-a2e3-7cab7531ef7a-kube-api-access-z9l7s\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:17.903020 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:17.902965 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jvnss\" (UniqueName: \"kubernetes.io/projected/b447616e-0dc2-4299-b91a-e4b22ee30867-kube-api-access-jvnss\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:18.030173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.030124 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:30:18.030959 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.030691 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" containerID="cri-o://fb8112cc159faaf807e0854ed42adacaab68be4661f3c0680696f911a21c045c" gracePeriod=30 Oct 31 00:30:18.034915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.034259 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:30:18.034915 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.034569 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" containerID="cri-o://69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7" gracePeriod=30 Oct 31 00:30:18.083214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.083179 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" path="/var/lib/kubelet/pods/9c514e69-2fcc-4c62-a390-eddc0863076d/volumes" Oct 31 00:30:18.084583 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.084557 6234 scope.go:122] "RemoveContainer" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" Oct 31 00:30:18.385631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.385592 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" Oct 31 00:30:18.385631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.385611 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv" event={"ID":"6773cff0-24af-4eaf-a2e3-7cab7531ef7a","Type":"ContainerDied","Data":"e130aae0bfb58efd605e0e343160518c552f10c324413bc1ec5c243c0c1abf8a"} Oct 31 00:30:18.386302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.385677 6234 scope.go:122] "RemoveContainer" containerID="4320522dcf4ffe3063c80daf296fde07fee79c90106a84011a4b3c453368ef01" Oct 31 00:30:18.387498 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.387472 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" event={"ID":"b447616e-0dc2-4299-b91a-e4b22ee30867","Type":"ContainerDied","Data":"351be2dff179808b2429836cfaf6afa5b6a6fbc51e689e63571bc314b412a020"} Oct 31 00:30:18.387642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.387570 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp" Oct 31 00:30:18.396452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.396427 6234 scope.go:122] "RemoveContainer" containerID="ed1c874b0361df2805b6f8f124f767cd5b5b3b489b061b6130f884244da4a30d" Oct 31 00:30:18.415668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.415640 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:18.443892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.443864 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-7b2341-29890110-n6xrv"] Oct 31 00:30:18.502605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.502560 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:18.540958 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:18.540877 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-14e39b-29890110-6jfzp"] Oct 31 00:30:19.406341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:19.406210 6234 generic.go:356] "Generic (PLEG): container finished" podID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerID="565f2dc7291536ff339a6beff416fdd52258aa8a63b229ef9dd7b0a5a70fb6e9" exitCode=60 Oct 31 00:30:19.406879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:19.406242 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerDied","Data":"565f2dc7291536ff339a6beff416fdd52258aa8a63b229ef9dd7b0a5a70fb6e9"} Oct 31 00:30:19.406879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:19.406519 6234 scope.go:122] "RemoveContainer" containerID="ad9373b568396e63771dac377b49ccdd1a5b73fb1578225f1b52a352a6ff8bfc" Oct 31 00:30:19.407238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:19.407212 6234 scope.go:122] "RemoveContainer" containerID="565f2dc7291536ff339a6beff416fdd52258aa8a63b229ef9dd7b0a5a70fb6e9" Oct 31 00:30:19.407701 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:19.407676 6234 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx_odh-ai-gateway-infra(da0eacb9-63e0-4721-b3a1-d91e386cccbe)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" Oct 31 00:30:20.082580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.082548 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" path="/var/lib/kubelet/pods/6773cff0-24af-4eaf-a2e3-7cab7531ef7a/volumes" Oct 31 00:30:20.083062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.083046 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" path="/var/lib/kubelet/pods/b447616e-0dc2-4299-b91a-e4b22ee30867/volumes" Oct 31 00:30:20.459699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.459661 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:20.715458 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.715401 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:20.763853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.763817 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\") pod \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " Oct 31 00:30:20.764091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.764069 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\" (UniqueName: \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\") pod \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\" (UID: \"da0eacb9-63e0-4721-b3a1-d91e386cccbe\") " Oct 31 00:30:20.764252 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.764222 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca" pod "da0eacb9-63e0-4721-b3a1-d91e386cccbe" (UID: "da0eacb9-63e0-4721-b3a1-d91e386cccbe"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:20.764770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.764748 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/da0eacb9-63e0-4721-b3a1-d91e386cccbe-maas-api-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:20.767093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.767070 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k" pod "da0eacb9-63e0-4721-b3a1-d91e386cccbe" (UID: "da0eacb9-63e0-4721-b3a1-d91e386cccbe"). InnerVolumeSpecName "kube-api-access-gbg7k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:20.866292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:20.866259 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gbg7k\" (UniqueName: \"kubernetes.io/projected/da0eacb9-63e0-4721-b3a1-d91e386cccbe-kube-api-access-gbg7k\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:21.426997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:21.426971 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" Oct 31 00:30:21.427163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:21.426966 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx" event={"ID":"da0eacb9-63e0-4721-b3a1-d91e386cccbe","Type":"ContainerDied","Data":"871f2c5ccfea1cdfd508f48e0ca3e0197fc87fe8a5945b43dfde14d78c8facef"} Oct 31 00:30:21.427163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:21.427108 6234 scope.go:122] "RemoveContainer" containerID="565f2dc7291536ff339a6beff416fdd52258aa8a63b229ef9dd7b0a5a70fb6e9" Oct 31 00:30:21.464450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:21.464418 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:21.491425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:21.491393 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-api-b-0d4ad9-29890110-d8cnx"] Oct 31 00:30:22.070945 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:22.070913 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z"] Oct 31 00:30:22.082762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:22.082732 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" path="/var/lib/kubelet/pods/da0eacb9-63e0-4721-b3a1-d91e386cccbe/volumes" Oct 31 00:30:22.086262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:22.086236 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:30:22.090496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:22.090467 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-d91de6-tjb6z"] Oct 31 00:30:23.314245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.314175 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417288 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417364 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417395 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417578 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417624 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417726 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417951 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.417996 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.422019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.418025 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\") pod \"8c4382fe-0417-4b17-be3f-b4bd257b8876\" (UID: \"8c4382fe-0417-4b17-be3f-b4bd257b8876\") " Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.431907 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.432132 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.432464 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.438329 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.438557 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.440160 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.440915 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.456931 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "kube-api-access-lmdkw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.475936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.461378 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo" pod "8c4382fe-0417-4b17-be3f-b4bd257b8876" (UID: "8c4382fe-0417-4b17-be3f-b4bd257b8876"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:23.483379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.483344 6234 generic.go:356] "Generic (PLEG): container finished" podID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerID="69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7" exitCode=0 Oct 31 00:30:23.483514 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.483501 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" event={"ID":"8c4382fe-0417-4b17-be3f-b4bd257b8876","Type":"ContainerDied","Data":"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7"} Oct 31 00:30:23.483585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.483570 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" event={"ID":"8c4382fe-0417-4b17-be3f-b4bd257b8876","Type":"ContainerDied","Data":"278a71a5e2c6b461fc75ee74c3804577daee5d5d5075a551b195b9b306b8bfb4"} Oct 31 00:30:23.483633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.483604 6234 scope.go:122] "RemoveContainer" containerID="69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7" Oct 31 00:30:23.483753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.483741 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk" Oct 31 00:30:23.508357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.508324 6234 generic.go:356] "Generic (PLEG): container finished" podID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerID="fb8112cc159faaf807e0854ed42adacaab68be4661f3c0680696f911a21c045c" exitCode=0 Oct 31 00:30:23.508525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.508507 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerDied","Data":"fb8112cc159faaf807e0854ed42adacaab68be4661f3c0680696f911a21c045c"} Oct 31 00:30:23.508627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.508566 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" event={"ID":"b993a36b-71fe-4361-a86d-fd8e6b4bd396","Type":"ContainerDied","Data":"f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126"} Oct 31 00:30:23.508627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.508594 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f9b714cab06bc4bc54e550e1e3e099b9d09f73fb0abbbd0de9690670a0cbc126" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532123 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532158 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmdkw\" (UniqueName: \"kubernetes.io/projected/8c4382fe-0417-4b17-be3f-b4bd257b8876-kube-api-access-lmdkw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532179 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532193 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532208 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532222 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532235 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8c4382fe-0417-4b17-be3f-b4bd257b8876-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532249 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.534841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.532263 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8c4382fe-0417-4b17-be3f-b4bd257b8876-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.585106 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.578650 6234 scope.go:122] "RemoveContainer" containerID="69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7" Oct 31 00:30:23.587235 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:23.585693 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7\": container with ID starting with 69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7 not found: ID does not exist" containerID="69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7" Oct 31 00:30:23.587235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.585741 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7"} err="failed to get container status \"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7\": rpc error: code = NotFound desc = could not find container \"69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7\": container with ID starting with 69659b01ea7740b7e211ff0d516561bd16b793fb8c80940713d9ce598c8794b7 not found: ID does not exist" Oct 31 00:30:23.587235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.585768 6234 scope.go:122] "RemoveContainer" containerID="6ba28c750492180b363d7baaaf2a98d9826621b7ca6408c9955c01bd50b7bb0b" Oct 31 00:30:23.592602 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.592569 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:30:23.686636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.686601 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:30:23.694504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.694462 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-lvrbk"] Oct 31 00:30:23.734934 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.734895 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.734970 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735027 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735080 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735147 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735204 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735422 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735394 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735453 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735538 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\") pod \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\" (UID: \"b993a36b-71fe-4361-a86d-fd8e6b4bd396\") " Oct 31 00:30:23.735625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735576 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.735625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735592 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:23.735829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.735808 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.736033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736010 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.736140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736014 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.736234 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736189 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.736347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736331 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.736610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736523 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.736610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.736610 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.737540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.737518 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.737862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.737845 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:23.738073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.738050 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "kube-api-access-p5tbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:23.738358 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.738343 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy" pod "b993a36b-71fe-4361-a86d-fd8e6b4bd396" (UID: "b993a36b-71fe-4361-a86d-fd8e6b4bd396"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:23.838293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.838212 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p5tbx\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-kube-api-access-p5tbx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.838293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.838244 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.838293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.838255 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.838293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.838264 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:23.838293 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:23.838273 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/b993a36b-71fe-4361-a86d-fd8e6b4bd396-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:24.096583 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:24.096495 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" path="/var/lib/kubelet/pods/8c4382fe-0417-4b17-be3f-b4bd257b8876/volumes" Oct 31 00:30:24.097537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:24.097511 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8f437e9e-96fe-45ba-8757-c62b0e93319a" path="/var/lib/kubelet/pods/8f437e9e-96fe-45ba-8757-c62b0e93319a/volumes" Oct 31 00:30:24.517271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:24.517244 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n" Oct 31 00:30:24.584460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:24.584420 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:30:24.590654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:24.590627 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-4c360e-openshift-default-78775bdc49-nxj9n"] Oct 31 00:30:26.082194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:26.082157 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" path="/var/lib/kubelet/pods/b993a36b-71fe-4361-a86d-fd8e6b4bd396/volumes" Oct 31 00:30:27.586772 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:27.586735 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:30:27.601941 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:27.601906 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:30:29.802260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:29.802196 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" containerID="cri-o://5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644" gracePeriod=45 Oct 31 00:30:29.876331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:29.876258 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" containerID="cri-o://4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9" gracePeriod=45 Oct 31 00:30:29.995425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:29.995154 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.78:8000/ready\": dial tcp 10.134.0.78:8000: connect: connection refused" Oct 31 00:30:30.191079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.191045 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:30:30.257946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.257914 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:30:30.302093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.302068 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:30:30.356894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.356759 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.356894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.356863 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\" (UniqueName: \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.356905 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\" (UniqueName: \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.357130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357012 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357054 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.357130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357069 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.357130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357114 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357152 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\" (UniqueName: \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357221 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\") pod \"e07e4d29-303a-4c97-8715-e9804db8d803\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357295 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357308 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357341 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357371 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\" (UniqueName: \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\") pod \"e07e4d29-303a-4c97-8715-e9804db8d803\" (UID: \"e07e4d29-303a-4c97-8715-e9804db8d803\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357423 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357461 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\") pod \"719d7203-6520-42c5-9c07-0837c6561a8d\" (UID: \"719d7203-6520-42c5-9c07-0837c6561a8d\") " Oct 31 00:30:30.357963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357507 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.357963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357528 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\") pod \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\" (UID: \"c6d115f3-0494-4a5d-bf51-d3358c96c1b9\") " Oct 31 00:30:30.357963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.357648 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume" pod "e07e4d29-303a-4c97-8715-e9804db8d803" (UID: "e07e4d29-303a-4c97-8715-e9804db8d803"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:30.358213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358186 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.358311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358219 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.358311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358234 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.358311 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358248 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e07e4d29-303a-4c97-8715-e9804db8d803-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.358637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358610 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.358861 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358810 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.358969 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.358946 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.360642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.360612 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.360775 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.360728 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "kube-api-access-p7zt7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.361175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.361032 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:30.361421 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.361398 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:30.361822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.361753 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6" pod "719d7203-6520-42c5-9c07-0837c6561a8d" (UID: "719d7203-6520-42c5-9c07-0837c6561a8d"). InnerVolumeSpecName "kube-api-access-rmml6". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.362687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.362222 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs" pod "c6d115f3-0494-4a5d-bf51-d3358c96c1b9" (UID: "c6d115f3-0494-4a5d-bf51-d3358c96c1b9"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:30.362687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.362314 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs" pod "e07e4d29-303a-4c97-8715-e9804db8d803" (UID: "e07e4d29-303a-4c97-8715-e9804db8d803"). InnerVolumeSpecName "kube-api-access-sgmxs". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:30.459107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459078 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459104 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459113 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgmxs\" (UniqueName: \"kubernetes.io/projected/e07e4d29-303a-4c97-8715-e9804db8d803-kube-api-access-sgmxs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459124 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459133 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/719d7203-6520-42c5-9c07-0837c6561a8d-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459143 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459153 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7zt7\" (UniqueName: \"kubernetes.io/projected/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-kube-api-access-p7zt7\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459162 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/719d7203-6520-42c5-9c07-0837c6561a8d-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459170 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/c6d115f3-0494-4a5d-bf51-d3358c96c1b9-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.459343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.459178 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rmml6\" (UniqueName: \"kubernetes.io/projected/719d7203-6520-42c5-9c07-0837c6561a8d-kube-api-access-rmml6\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:30.592724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.592693 6234 generic.go:356] "Generic (PLEG): container finished" podID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerID="5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644" exitCode=0 Oct 31 00:30:30.592946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.592740 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" Oct 31 00:30:30.592946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.592793 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" event={"ID":"c6d115f3-0494-4a5d-bf51-d3358c96c1b9","Type":"ContainerDied","Data":"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644"} Oct 31 00:30:30.592946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.592838 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58" event={"ID":"c6d115f3-0494-4a5d-bf51-d3358c96c1b9","Type":"ContainerDied","Data":"4169e7aea03cbdd363dccd339fff83a886826b8dd2964150ef45e20cf77571e4"} Oct 31 00:30:30.592946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.592862 6234 scope.go:122] "RemoveContainer" containerID="5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644" Oct 31 00:30:30.594377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.594353 6234 generic.go:356] "Generic (PLEG): container finished" podID="719d7203-6520-42c5-9c07-0837c6561a8d" containerID="4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9" exitCode=0 Oct 31 00:30:30.594478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.594398 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" Oct 31 00:30:30.594478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.594400 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" event={"ID":"719d7203-6520-42c5-9c07-0837c6561a8d","Type":"ContainerDied","Data":"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9"} Oct 31 00:30:30.594478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.594443 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2" event={"ID":"719d7203-6520-42c5-9c07-0837c6561a8d","Type":"ContainerDied","Data":"35605ffe6d8851d580f98e0eef00657f84aced78389afa9a749e4ad56357a7c2"} Oct 31 00:30:30.596223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.596198 6234 generic.go:356] "Generic (PLEG): container finished" podID="e07e4d29-303a-4c97-8715-e9804db8d803" containerID="ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24" exitCode=0 Oct 31 00:30:30.596322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.596240 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" Oct 31 00:30:30.596322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.596268 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" event={"ID":"e07e4d29-303a-4c97-8715-e9804db8d803","Type":"ContainerDied","Data":"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24"} Oct 31 00:30:30.596322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.596305 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t" event={"ID":"e07e4d29-303a-4c97-8715-e9804db8d803","Type":"ContainerDied","Data":"76c708489d048b4ab5b920de894760378d16d93e1da3e69ad39cd14fbd173cce"} Oct 31 00:30:30.603875 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.603857 6234 scope.go:122] "RemoveContainer" containerID="5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644" Oct 31 00:30:30.604383 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:30.604195 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644\": container with ID starting with 5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644 not found: ID does not exist" containerID="5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644" Oct 31 00:30:30.604383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.604228 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644"} err="failed to get container status \"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644\": rpc error: code = NotFound desc = could not find container \"5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644\": container with ID starting with 5ac91e8a018e8840ead255a8d652763eebc04932babda4666da2587b5d008644 not found: ID does not exist" Oct 31 00:30:30.604383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.604250 6234 scope.go:122] "RemoveContainer" containerID="4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9" Oct 31 00:30:30.615209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.615189 6234 scope.go:122] "RemoveContainer" containerID="4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9" Oct 31 00:30:30.615520 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:30.615500 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9\": container with ID starting with 4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9 not found: ID does not exist" containerID="4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9" Oct 31 00:30:30.615595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.615529 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9"} err="failed to get container status \"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9\": rpc error: code = NotFound desc = could not find container \"4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9\": container with ID starting with 4e667997864c4dd3b8462b90fd9fe6d1b801154847361c2629aa3abb17544ca9 not found: ID does not exist" Oct 31 00:30:30.615595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.615544 6234 scope.go:122] "RemoveContainer" containerID="ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24" Oct 31 00:30:30.625010 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.624992 6234 scope.go:122] "RemoveContainer" containerID="ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24" Oct 31 00:30:30.625272 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:30.625252 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24\": container with ID starting with ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24 not found: ID does not exist" containerID="ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24" Oct 31 00:30:30.625314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.625281 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24"} err="failed to get container status \"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24\": rpc error: code = NotFound desc = could not find container \"ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24\": container with ID starting with ea65f719540aa95d7c6c2e3745f04d965947dbbb1ea4fadd2bdc8a46abc92b24 not found: ID does not exist" Oct 31 00:30:30.639062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.639032 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:30:30.647753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.647730 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/distinct-2-w0-c60826-kserve-b55d899-brb58"] Oct 31 00:30:30.675893 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.675849 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:30:30.677911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.677892 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-d91de6-758c465cf-5vv6t"] Oct 31 00:30:30.690108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.690079 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:30:30.696170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.696146 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c60826/simulator-w0-c60826-kserve-5d6dd5b68-6pnk2"] Oct 31 00:30:30.891297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.891213 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:30:30.891760 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.891413 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" containerID="cri-o://7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba" gracePeriod=30 Oct 31 00:30:30.933701 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.933663 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:30:30.933937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:30.933914 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" containerID="cri-o://c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e" gracePeriod=30 Oct 31 00:30:32.083297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.083265 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" path="/var/lib/kubelet/pods/719d7203-6520-42c5-9c07-0837c6561a8d/volumes" Oct 31 00:30:32.083721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.083707 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" path="/var/lib/kubelet/pods/c6d115f3-0494-4a5d-bf51-d3358c96c1b9/volumes" Oct 31 00:30:32.084127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.084113 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" path="/var/lib/kubelet/pods/e07e4d29-303a-4c97-8715-e9804db8d803/volumes" Oct 31 00:30:32.587713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.587666 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" podUID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" containerName="payload-processing" containerID="cri-o://bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d" gracePeriod=25 Oct 31 00:30:32.602442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.602405 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" podUID="f8eda208-4692-45bd-ba91-9eb470e5104c" containerName="payload-pre-processing" containerID="cri-o://c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487" gracePeriod=25 Oct 31 00:30:32.874650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.874623 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:30:32.908635 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.908608 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:30:32.914584 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.914561 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\") pod \"f8eda208-4692-45bd-ba91-9eb470e5104c\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " Oct 31 00:30:32.914716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.914602 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\") pod \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " Oct 31 00:30:32.914716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.914663 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\" (UniqueName: \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\") pod \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\" (UID: \"639d6597-1d61-4757-8dfe-2e4ddf1e57f3\") " Oct 31 00:30:32.914890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.914840 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\" (UniqueName: \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\") pod \"f8eda208-4692-45bd-ba91-9eb470e5104c\" (UID: \"f8eda208-4692-45bd-ba91-9eb470e5104c\") " Oct 31 00:30:32.915042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.915019 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume" pod "639d6597-1d61-4757-8dfe-2e4ddf1e57f3" (UID: "639d6597-1d61-4757-8dfe-2e4ddf1e57f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:32.915100 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.915016 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume" pod "f8eda208-4692-45bd-ba91-9eb470e5104c" (UID: "f8eda208-4692-45bd-ba91-9eb470e5104c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:32.915370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.915351 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f8eda208-4692-45bd-ba91-9eb470e5104c-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:32.915463 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.915376 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:32.917164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.917130 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5" pod "639d6597-1d61-4757-8dfe-2e4ddf1e57f3" (UID: "639d6597-1d61-4757-8dfe-2e4ddf1e57f3"). InnerVolumeSpecName "kube-api-access-g6fm5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:32.917254 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:32.917177 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt" pod "f8eda208-4692-45bd-ba91-9eb470e5104c" (UID: "f8eda208-4692-45bd-ba91-9eb470e5104c"). InnerVolumeSpecName "kube-api-access-j66nt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:33.016559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.016519 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g6fm5\" (UniqueName: \"kubernetes.io/projected/639d6597-1d61-4757-8dfe-2e4ddf1e57f3-kube-api-access-g6fm5\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:33.016559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.016556 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j66nt\" (UniqueName: \"kubernetes.io/projected/f8eda208-4692-45bd-ba91-9eb470e5104c-kube-api-access-j66nt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:33.161545 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:33.161460 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:30:33.621278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.621240 6234 generic.go:356] "Generic (PLEG): container finished" podID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" containerID="bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d" exitCode=0 Oct 31 00:30:33.621438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.621289 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" Oct 31 00:30:33.621438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.621320 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" event={"ID":"639d6597-1d61-4757-8dfe-2e4ddf1e57f3","Type":"ContainerDied","Data":"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d"} Oct 31 00:30:33.621438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.621367 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5" event={"ID":"639d6597-1d61-4757-8dfe-2e4ddf1e57f3","Type":"ContainerDied","Data":"c52a6c5b9d8afa21c9c03fde5ae1e73c9b1eb41178939d33253d8b8ea1237f9e"} Oct 31 00:30:33.621438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.621397 6234 scope.go:122] "RemoveContainer" containerID="bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d" Oct 31 00:30:33.622873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.622853 6234 generic.go:356] "Generic (PLEG): container finished" podID="f8eda208-4692-45bd-ba91-9eb470e5104c" containerID="c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487" exitCode=0 Oct 31 00:30:33.622964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.622880 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" event={"ID":"f8eda208-4692-45bd-ba91-9eb470e5104c","Type":"ContainerDied","Data":"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487"} Oct 31 00:30:33.622964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.622918 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" Oct 31 00:30:33.622964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.622935 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm" event={"ID":"f8eda208-4692-45bd-ba91-9eb470e5104c","Type":"ContainerDied","Data":"b1efb3e989c73d387e9d39896ede2048d098ccec0c7de13e2fa4056b8af21654"} Oct 31 00:30:33.638152 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.638126 6234 scope.go:122] "RemoveContainer" containerID="bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d" Oct 31 00:30:33.638542 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:33.638516 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d\": container with ID starting with bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d not found: ID does not exist" containerID="bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d" Oct 31 00:30:33.638647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.638550 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d"} err="failed to get container status \"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d\": rpc error: code = NotFound desc = could not find container \"bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d\": container with ID starting with bc92678851d0a0eedfa169822029741aba63de2e57a95a46337f04ec8e594d7d not found: ID does not exist" Oct 31 00:30:33.638647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.638568 6234 scope.go:122] "RemoveContainer" containerID="c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487" Oct 31 00:30:33.651633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.651611 6234 scope.go:122] "RemoveContainer" containerID="c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487" Oct 31 00:30:33.652036 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:33.651998 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487\": container with ID starting with c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487 not found: ID does not exist" containerID="c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487" Oct 31 00:30:33.652209 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.652037 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487"} err="failed to get container status \"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487\": rpc error: code = NotFound desc = could not find container \"c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487\": container with ID starting with c37f9e13ecf522b39991c50a90c6e4236618d80962f0425587cb2ad49dfac487 not found: ID does not exist" Oct 31 00:30:33.657228 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.657206 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:30:33.663548 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.663523 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-31108d-5f9b64b99-c8ht5"] Oct 31 00:30:33.676157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.676132 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:30:33.684230 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:33.684203 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-31108d-6757596d7-rsqrm"] Oct 31 00:30:34.083981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:34.083940 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" path="/var/lib/kubelet/pods/639d6597-1d61-4757-8dfe-2e4ddf1e57f3/volumes" Oct 31 00:30:34.084599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:34.084579 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f8eda208-4692-45bd-ba91-9eb470e5104c" path="/var/lib/kubelet/pods/f8eda208-4692-45bd-ba91-9eb470e5104c/volumes" Oct 31 00:30:36.250627 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.250598 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:30:36.291352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.291322 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:30:36.358926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.358894 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.358938 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.358985 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359026 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359081 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359117 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359175 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359202 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359274 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359334 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359385 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359419 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359489 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359557 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.359718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359591 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.359718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359619 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.360016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.359747 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.360576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.360103 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\") pod \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\" (UID: \"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc\") " Oct 31 00:30:36.360576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.360162 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\") pod \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\" (UID: \"d6a93d93-30dc-43fc-a3a3-b640229dfc1f\") " Oct 31 00:30:36.360576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.360259 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:36.360839 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.360575 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.362184 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.361990 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "kube-api-access-c8nhk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.362562 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.362320 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.362699 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.362631 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.362837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.362814 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.363092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.362991 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.363092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363012 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.363266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363185 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.363863 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363836 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363865 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363880 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363893 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363907 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363920 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c8nhk\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-kube-api-access-c8nhk\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363935 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363949 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.363966 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363963 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.364401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363978 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.364401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.363990 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.364401 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.364045 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:36.364876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.364835 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:36.364973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.364932 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "kube-api-access-hh4fb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.365322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.365296 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.365412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.365384 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo" pod "d6a93d93-30dc-43fc-a3a3-b640229dfc1f" (UID: "d6a93d93-30dc-43fc-a3a3-b640229dfc1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:36.365412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.365403 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:36.366008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.365987 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token" pod "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" (UID: "8e2ec100-4d8c-4cce-974c-18ab8b60f3cc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465055 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465092 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465101 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465111 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465122 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hh4fb\" (UniqueName: \"kubernetes.io/projected/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-kube-api-access-hh4fb\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465131 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.465140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.465140 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d6a93d93-30dc-43fc-a3a3-b640229dfc1f-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:36.651355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.651318 6234 generic.go:356] "Generic (PLEG): container finished" podID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerID="c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e" exitCode=0 Oct 31 00:30:36.651550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.651367 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" Oct 31 00:30:36.651550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.651401 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" event={"ID":"d6a93d93-30dc-43fc-a3a3-b640229dfc1f","Type":"ContainerDied","Data":"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e"} Oct 31 00:30:36.651550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.651450 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j" event={"ID":"d6a93d93-30dc-43fc-a3a3-b640229dfc1f","Type":"ContainerDied","Data":"3a2844622ea69b226fe604f87675b9ae8fb6ea05909a65dd0430b9ea906e2d88"} Oct 31 00:30:36.651550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.651479 6234 scope.go:122] "RemoveContainer" containerID="c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e" Oct 31 00:30:36.653274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.653244 6234 generic.go:356] "Generic (PLEG): container finished" podID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerID="7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba" exitCode=0 Oct 31 00:30:36.653372 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.653325 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" Oct 31 00:30:36.653430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.653380 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" event={"ID":"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc","Type":"ContainerDied","Data":"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba"} Oct 31 00:30:36.653491 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.653426 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr" event={"ID":"8e2ec100-4d8c-4cce-974c-18ab8b60f3cc","Type":"ContainerDied","Data":"d6966a92ac743243492d3cb311e967b8c0d5a718fb85eaa1c477463d715ff1b9"} Oct 31 00:30:36.663481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.663460 6234 scope.go:122] "RemoveContainer" containerID="c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e" Oct 31 00:30:36.663753 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:36.663735 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e\": container with ID starting with c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e not found: ID does not exist" containerID="c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e" Oct 31 00:30:36.663862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.663761 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e"} err="failed to get container status \"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e\": rpc error: code = NotFound desc = could not find container \"c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e\": container with ID starting with c76b7a8028a17a3f6ff041cdcb113348d6a440dfcf83d8f4d69192e084bc423e not found: ID does not exist" Oct 31 00:30:36.663862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.663793 6234 scope.go:122] "RemoveContainer" containerID="7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba" Oct 31 00:30:36.673314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.673290 6234 scope.go:122] "RemoveContainer" containerID="7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba" Oct 31 00:30:36.673653 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:36.673632 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba\": container with ID starting with 7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba not found: ID does not exist" containerID="7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba" Oct 31 00:30:36.673744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.673659 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba"} err="failed to get container status \"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba\": rpc error: code = NotFound desc = could not find container \"7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba\": container with ID starting with 7f763a961bf95ad4cff61f34589392be53a43424674ceda6198f6aeed522dfba not found: ID does not exist" Oct 31 00:30:36.689853 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.689827 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:30:36.694532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.694508 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-6jf5j"] Oct 31 00:30:36.708608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.708586 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:30:36.712420 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:36.712400 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-d91de6-openshift-default-587b6bd7dc-w27nr"] Oct 31 00:30:38.082855 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:38.082820 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" path="/var/lib/kubelet/pods/8e2ec100-4d8c-4cce-974c-18ab8b60f3cc/volumes" Oct 31 00:30:38.083302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:38.083288 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" path="/var/lib/kubelet/pods/d6a93d93-30dc-43fc-a3a3-b640229dfc1f/volumes" Oct 31 00:30:40.723638 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:40.723602 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:30:41.051864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.048967 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:30:41.077814 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.063083 6234 scope.go:122] "RemoveContainer" containerID="fb8112cc159faaf807e0854ed42adacaab68be4661f3c0680696f911a21c045c" Oct 31 00:30:41.440833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.438281 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:30:41.440833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.438355 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:30:41.440833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.438737 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:30:41.440833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:41.438807 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:30:43.132003 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:43.131962 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:30:43.139626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:43.139595 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:30:45.175121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.175086 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:30:45.175508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.175286 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" podUID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" containerName="maas-api" containerID="cri-o://3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71" gracePeriod=30 Oct 31 00:30:45.566002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.565975 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:30:45.686750 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.686719 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") pod \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " Oct 31 00:30:45.686947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.686759 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\") pod \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " Oct 31 00:30:45.686947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.686868 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\") pod \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " Oct 31 00:30:45.687034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.686961 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\" (UniqueName: \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\") pod \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\" (UID: \"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f\") " Oct 31 00:30:45.687304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.687272 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca" pod "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" (UID: "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:45.687487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.687467 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:45.689245 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.689214 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd" pod "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" (UID: "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f"). InnerVolumeSpecName "kube-api-access-2vrbd". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:45.689583 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.689562 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls" pod "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" (UID: "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:45.689658 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.689577 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs" pod "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" (UID: "7b85dda1-ff52-42e7-b9fd-f54f6fcb803f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:45.728731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.727898 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" podUID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" containerName="payload-processing" containerID="cri-o://15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47" gracePeriod=25 Oct 31 00:30:45.789058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.788913 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vrbd\" (UniqueName: \"kubernetes.io/projected/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-kube-api-access-2vrbd\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:45.789058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.788946 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:45.789058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.788958 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:45.996765 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:45.996739 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:30:46.024843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.024776 6234 generic.go:356] "Generic (PLEG): container finished" podID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" containerID="15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47" exitCode=0 Oct 31 00:30:46.024843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.024834 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" Oct 31 00:30:46.025054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.024861 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" event={"ID":"374240e2-0c6a-4ec6-9db8-b07d279a49ad","Type":"ContainerDied","Data":"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47"} Oct 31 00:30:46.025054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.024920 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq" event={"ID":"374240e2-0c6a-4ec6-9db8-b07d279a49ad","Type":"ContainerDied","Data":"7e2ede2d083a8bd2e68555bb74dcf6e596daac84e70509d1b4a8dbf31c57c0eb"} Oct 31 00:30:46.025054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.024948 6234 scope.go:122] "RemoveContainer" containerID="15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47" Oct 31 00:30:46.026262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.026237 6234 generic.go:356] "Generic (PLEG): container finished" podID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" containerID="3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71" exitCode=0 Oct 31 00:30:46.026384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.026271 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" event={"ID":"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f","Type":"ContainerDied","Data":"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71"} Oct 31 00:30:46.026384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.026312 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" event={"ID":"7b85dda1-ff52-42e7-b9fd-f54f6fcb803f","Type":"ContainerDied","Data":"321bdd3f2489b56e6863fbd5c89c2aa6bbf7ab9b05b3fa3b0ed9f05b03120b9e"} Oct 31 00:30:46.026384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.026367 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r" Oct 31 00:30:46.039478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.039416 6234 scope.go:122] "RemoveContainer" containerID="15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47" Oct 31 00:30:46.039888 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:46.039864 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47\": container with ID starting with 15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47 not found: ID does not exist" containerID="15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47" Oct 31 00:30:46.039990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.039894 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47"} err="failed to get container status \"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47\": rpc error: code = NotFound desc = could not find container \"15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47\": container with ID starting with 15e71f96cf12318c6d8279335b6ea1f807c71828939783c92602496114e70e47 not found: ID does not exist" Oct 31 00:30:46.039990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.039909 6234 scope.go:122] "RemoveContainer" containerID="3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71" Oct 31 00:30:46.049873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.049836 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" podUID="73170cf3-9711-40a6-95a2-5e013a98bc12" containerName="payload-pre-processing" containerID="cri-o://bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb" gracePeriod=25 Oct 31 00:30:46.051530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.051297 6234 scope.go:122] "RemoveContainer" containerID="3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71" Oct 31 00:30:46.051654 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:46.051613 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71\": container with ID starting with 3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71 not found: ID does not exist" containerID="3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71" Oct 31 00:30:46.051700 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.051644 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71"} err="failed to get container status \"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71\": rpc error: code = NotFound desc = could not find container \"3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71\": container with ID starting with 3503ff94a45df3f8016a908f3d11ae7b211608fa617b39d81fcd89cd0fa74f71 not found: ID does not exist" Oct 31 00:30:46.061193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.061163 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:30:46.068721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.068694 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w0-c60826-6f7987c7f6-f6n4r"] Oct 31 00:30:46.081454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.081431 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" path="/var/lib/kubelet/pods/7b85dda1-ff52-42e7-b9fd-f54f6fcb803f/volumes" Oct 31 00:30:46.092550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.092520 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\" (UniqueName: \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\") pod \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " Oct 31 00:30:46.092696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.092672 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\") pod \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\" (UID: \"374240e2-0c6a-4ec6-9db8-b07d279a49ad\") " Oct 31 00:30:46.093034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.093012 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume" pod "374240e2-0c6a-4ec6-9db8-b07d279a49ad" (UID: "374240e2-0c6a-4ec6-9db8-b07d279a49ad"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:46.093180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.093165 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/374240e2-0c6a-4ec6-9db8-b07d279a49ad-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:46.095177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.095157 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9" pod "374240e2-0c6a-4ec6-9db8-b07d279a49ad" (UID: "374240e2-0c6a-4ec6-9db8-b07d279a49ad"). InnerVolumeSpecName "kube-api-access-7dnh9". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:46.194899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.194860 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7dnh9\" (UniqueName: \"kubernetes.io/projected/374240e2-0c6a-4ec6-9db8-b07d279a49ad-kube-api-access-7dnh9\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:46.311074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.311048 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:30:46.350511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.350481 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:30:46.354247 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.354216 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-1c9660-5fd5f656c9-lr8mq"] Oct 31 00:30:46.397215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.397181 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\") pod \"73170cf3-9711-40a6-95a2-5e013a98bc12\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " Oct 31 00:30:46.397391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.397247 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\" (UniqueName: \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\") pod \"73170cf3-9711-40a6-95a2-5e013a98bc12\" (UID: \"73170cf3-9711-40a6-95a2-5e013a98bc12\") " Oct 31 00:30:46.397582 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.397556 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume" pod "73170cf3-9711-40a6-95a2-5e013a98bc12" (UID: "73170cf3-9711-40a6-95a2-5e013a98bc12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:46.397755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.397741 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/73170cf3-9711-40a6-95a2-5e013a98bc12-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:46.399486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.399460 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx" pod "73170cf3-9711-40a6-95a2-5e013a98bc12" (UID: "73170cf3-9711-40a6-95a2-5e013a98bc12"). InnerVolumeSpecName "kube-api-access-k7xbx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:46.499342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:46.499312 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7xbx\" (UniqueName: \"kubernetes.io/projected/73170cf3-9711-40a6-95a2-5e013a98bc12-kube-api-access-k7xbx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:47.034931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.034899 6234 generic.go:356] "Generic (PLEG): container finished" podID="73170cf3-9711-40a6-95a2-5e013a98bc12" containerID="bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb" exitCode=0 Oct 31 00:30:47.035093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.034955 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" Oct 31 00:30:47.035093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.034982 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" event={"ID":"73170cf3-9711-40a6-95a2-5e013a98bc12","Type":"ContainerDied","Data":"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb"} Oct 31 00:30:47.035093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.035036 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml" event={"ID":"73170cf3-9711-40a6-95a2-5e013a98bc12","Type":"ContainerDied","Data":"e9b616cc9d1bea0d2d868418dc2cd2442a9d39ca5175b06b455c670291971d5f"} Oct 31 00:30:47.035093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.035065 6234 scope.go:122] "RemoveContainer" containerID="bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb" Oct 31 00:30:47.045540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.045523 6234 scope.go:122] "RemoveContainer" containerID="bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb" Oct 31 00:30:47.045864 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:47.045843 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb\": container with ID starting with bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb not found: ID does not exist" containerID="bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb" Oct 31 00:30:47.045907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.045872 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb"} err="failed to get container status \"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb\": rpc error: code = NotFound desc = could not find container \"bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb\": container with ID starting with bd289b6725c3ca43de09e543374239fdb7c67806193896879eef1ff380ef72fb not found: ID does not exist" Oct 31 00:30:47.062974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.062947 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:30:47.068802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:47.068763 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-1c9660-5dcb5b7f59-z79ml"] Oct 31 00:30:48.081616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:48.081540 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" path="/var/lib/kubelet/pods/374240e2-0c6a-4ec6-9db8-b07d279a49ad/volumes" Oct 31 00:30:48.082073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:48.082056 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73170cf3-9711-40a6-95a2-5e013a98bc12" path="/var/lib/kubelet/pods/73170cf3-9711-40a6-95a2-5e013a98bc12/volumes" Oct 31 00:30:48.484886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:48.484842 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:30:50.290686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:50.290652 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:30:50.291158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:50.290936 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" containerID="cri-o://a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061" gracePeriod=30 Oct 31 00:30:53.485490 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:53.485442 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" containerID="cri-o://9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a" gracePeriod=25 Oct 31 00:30:54.035381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:54.035348 6234 patch_prober.go:29] interesting pod/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:30:54.035553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:54.035418 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:30:54.113647 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:54.113359 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:30:55.573887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.573861 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:30:55.629466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629435 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629481 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629511 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629544 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629573 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629654 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.629890 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629714 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.630013 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.629882 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:55.630174 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630150 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.630285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630168 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:55.630285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630180 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:55.630402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630323 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\") pod \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\" (UID: \"d9a8230f-61fd-42ed-aa40-b9fc374be85e\") " Oct 31 00:30:55.630631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630609 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:55.630988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.630962 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:30:55.631140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.631118 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.631204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.631147 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.631204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.631163 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.631204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.631177 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.631204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.631190 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.632103 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.632079 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:55.632332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.632304 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:30:55.632536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.632512 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "kube-api-access-rpwnc". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:55.633287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.633268 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token" pod "d9a8230f-61fd-42ed-aa40-b9fc374be85e" (UID: "d9a8230f-61fd-42ed-aa40-b9fc374be85e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:55.732662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.732620 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.732662 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.732661 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.732951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.732674 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rpwnc\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-kube-api-access-rpwnc\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:55.732951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:55.732690 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9a8230f-61fd-42ed-aa40-b9fc374be85e-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:56.130116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.130086 6234 generic.go:356] "Generic (PLEG): container finished" podID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerID="a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061" exitCode=0 Oct 31 00:30:56.130265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.130139 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" Oct 31 00:30:56.130265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.130178 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerDied","Data":"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061"} Oct 31 00:30:56.130265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.130234 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5" event={"ID":"d9a8230f-61fd-42ed-aa40-b9fc374be85e","Type":"ContainerDied","Data":"f0d1efae3c48f3010aad52164ab3102ac2b35b7e52dca105f52c13d57be57483"} Oct 31 00:30:56.130265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.130263 6234 scope.go:122] "RemoveContainer" containerID="a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061" Oct 31 00:30:56.140052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.140029 6234 scope.go:122] "RemoveContainer" containerID="5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98" Oct 31 00:30:56.148857 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.148839 6234 scope.go:122] "RemoveContainer" containerID="a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061" Oct 31 00:30:56.149127 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:56.149105 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061\": container with ID starting with a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061 not found: ID does not exist" containerID="a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061" Oct 31 00:30:56.149212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.149134 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061"} err="failed to get container status \"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061\": rpc error: code = NotFound desc = could not find container \"a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061\": container with ID starting with a6c9edd2c6bf6b36e09845473db823147c3fd875c54aafe18e85c662aaecf061 not found: ID does not exist" Oct 31 00:30:56.149212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.149151 6234 scope.go:122] "RemoveContainer" containerID="5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98" Oct 31 00:30:56.149428 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:56.149399 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98\": container with ID starting with 5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98 not found: ID does not exist" containerID="5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98" Oct 31 00:30:56.149466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.149441 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98"} err="failed to get container status \"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98\": rpc error: code = NotFound desc = could not find container \"5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98\": container with ID starting with 5a0c4b25308d8578191dda01ffab0254341ae5b4205b851ae23d094c940b0e98 not found: ID does not exist" Oct 31 00:30:56.156995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.156974 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:30:56.166225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.166203 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-1c9660-openshift-default-69bcc84ddf-zp8z5"] Oct 31 00:30:56.925720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.925685 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:30:56.926133 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:56.925937 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" containerID="cri-o://c0f43c82964f9de5db2181d9315bd2ece5da5128fa2ca196db6df8b7986dbf6d" gracePeriod=30 Oct 31 00:30:58.081965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.081925 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" path="/var/lib/kubelet/pods/d9a8230f-61fd-42ed-aa40-b9fc374be85e/volumes" Oct 31 00:30:58.153909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.153866 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" containerID="cri-o://a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085" gracePeriod=45 Oct 31 00:30:58.160859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.160824 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" containerID="cri-o://5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c" gracePeriod=45 Oct 31 00:30:58.434062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.433799 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:30:58.457187 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.457161 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493508 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493559 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493608 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493709 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\" (UniqueName: \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493843 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\" (UniqueName: \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.496127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.493873 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\") pod \"75668942-ad30-4251-ad4b-b5df786ff9ae\" (UID: \"75668942-ad30-4251-ad4b-b5df786ff9ae\") " Oct 31 00:30:58.545109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.543805 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.547180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.546527 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.547180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.546887 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.550027 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.549998 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.551144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.551105 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:58.564073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.564042 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm" pod "75668942-ad30-4251-ad4b-b5df786ff9ae" (UID: "75668942-ad30-4251-ad4b-b5df786ff9ae"). InnerVolumeSpecName "kube-api-access-hzghm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:58.598116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598078 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\" (UniqueName: \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.598250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598191 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.598327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598311 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.598381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598344 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.598421 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598407 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.598522 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598497 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\" (UniqueName: \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\") pod \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\" (UID: \"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4\") " Oct 31 00:30:58.599019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.598997 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/75668942-ad30-4251-ad4b-b5df786ff9ae-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599027 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzghm\" (UniqueName: \"kubernetes.io/projected/75668942-ad30-4251-ad4b-b5df786ff9ae-kube-api-access-hzghm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599043 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599056 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599068 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599081 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/75668942-ad30-4251-ad4b-b5df786ff9ae-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.599907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599690 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.599907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.599691 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.600158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.600134 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.602041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.602019 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:30:58.602129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.602096 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "kube-api-access-8sf9g". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:30:58.602242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.602228 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm" pod "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" (UID: "3c13ba68-b7ce-40f4-8232-c8d2203cd6b4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:30:58.699898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699864 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.699898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699896 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.700198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699909 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.700198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699920 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.700198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699933 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:58.700198 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:58.699947 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8sf9g\" (UniqueName: \"kubernetes.io/projected/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4-kube-api-access-8sf9g\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:30:59.160774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.160689 6234 generic.go:356] "Generic (PLEG): container finished" podID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerID="5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c" exitCode=0 Oct 31 00:30:59.160774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.160752 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" Oct 31 00:30:59.161406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.160776 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" event={"ID":"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4","Type":"ContainerDied","Data":"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c"} Oct 31 00:30:59.161406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.160846 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv" event={"ID":"3c13ba68-b7ce-40f4-8232-c8d2203cd6b4","Type":"ContainerDied","Data":"0f1f0c8eca1e020b09014b5eb41cb8c8a622b5970714914e30b9bc8b97ab23a2"} Oct 31 00:30:59.161406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.160867 6234 scope.go:122] "RemoveContainer" containerID="5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c" Oct 31 00:30:59.162269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.162243 6234 generic.go:356] "Generic (PLEG): container finished" podID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerID="a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085" exitCode=0 Oct 31 00:30:59.162350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.162305 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" Oct 31 00:30:59.162411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.162353 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" event={"ID":"75668942-ad30-4251-ad4b-b5df786ff9ae","Type":"ContainerDied","Data":"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085"} Oct 31 00:30:59.162411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.162386 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p" event={"ID":"75668942-ad30-4251-ad4b-b5df786ff9ae","Type":"ContainerDied","Data":"a90c624f075f64b037295dcb96b3af693300e6be8ae23aed7c7aba79da3ddf0d"} Oct 31 00:30:59.185456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.185438 6234 scope.go:122] "RemoveContainer" containerID="5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c" Oct 31 00:30:59.185832 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:59.185775 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c\": container with ID starting with 5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c not found: ID does not exist" containerID="5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c" Oct 31 00:30:59.185938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.185838 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c"} err="failed to get container status \"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c\": rpc error: code = NotFound desc = could not find container \"5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c\": container with ID starting with 5e0153b44a4f96438390f096a08de5132e0179da82123b431ebd6034057bd38c not found: ID does not exist" Oct 31 00:30:59.185938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.185853 6234 scope.go:122] "RemoveContainer" containerID="a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085" Oct 31 00:30:59.195092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.195077 6234 scope.go:122] "RemoveContainer" containerID="a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085" Oct 31 00:30:59.195360 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:30:59.195342 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085\": container with ID starting with a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085 not found: ID does not exist" containerID="a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085" Oct 31 00:30:59.195430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.195365 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085"} err="failed to get container status \"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085\": rpc error: code = NotFound desc = could not find container \"a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085\": container with ID starting with a872fafb104f111897e06d335a4949098f659f00106e1bad9bb1c316191c9085 not found: ID does not exist" Oct 31 00:30:59.199276 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.199255 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:30:59.205354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.205334 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/distinct-w6-0402df-kserve-5c874c97f-nsk6p"] Oct 31 00:30:59.218225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.218202 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:30:59.222442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:30:59.222418 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w6-0402df/premium-w6-0402df-kserve-8d7957496-l48qv"] Oct 31 00:31:00.081161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:00.081128 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" path="/var/lib/kubelet/pods/3c13ba68-b7ce-40f4-8232-c8d2203cd6b4/volumes" Oct 31 00:31:00.081571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:00.081558 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" path="/var/lib/kubelet/pods/75668942-ad30-4251-ad4b-b5df786ff9ae/volumes" Oct 31 00:31:01.004545 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.004505 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:31:01.004961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.004766 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" podUID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" containerName="maas-api" containerID="cri-o://9cc5793cb30d5b34cafa7a3297140a92107fca2c13ddde391c4bfb1015dcc14c" gracePeriod=30 Oct 31 00:31:01.035640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.035610 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:31:01.177952 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.177908 6234 generic.go:356] "Generic (PLEG): container finished" podID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" containerID="9cc5793cb30d5b34cafa7a3297140a92107fca2c13ddde391c4bfb1015dcc14c" exitCode=0 Oct 31 00:31:01.178131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.178087 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" event={"ID":"47eb3460-dd38-4319-ba98-5ddf0c9011cf","Type":"ContainerDied","Data":"9cc5793cb30d5b34cafa7a3297140a92107fca2c13ddde391c4bfb1015dcc14c"} Oct 31 00:31:01.374808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.374035 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:31:01.432623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.432582 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\") pod \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " Oct 31 00:31:01.432840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.432676 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\") pod \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " Oct 31 00:31:01.432840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.432815 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\") pod \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " Oct 31 00:31:01.433055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.432861 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\" (UniqueName: \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\") pod \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\" (UID: \"47eb3460-dd38-4319-ba98-5ddf0c9011cf\") " Oct 31 00:31:01.434160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.433228 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca" pod "47eb3460-dd38-4319-ba98-5ddf0c9011cf" (UID: "47eb3460-dd38-4319-ba98-5ddf0c9011cf"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:01.434160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.433512 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/47eb3460-dd38-4319-ba98-5ddf0c9011cf-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:01.435425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.435401 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls" pod "47eb3460-dd38-4319-ba98-5ddf0c9011cf" (UID: "47eb3460-dd38-4319-ba98-5ddf0c9011cf"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:01.436910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.436651 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs" pod "47eb3460-dd38-4319-ba98-5ddf0c9011cf" (UID: "47eb3460-dd38-4319-ba98-5ddf0c9011cf"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:01.437007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.436965 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr" pod "47eb3460-dd38-4319-ba98-5ddf0c9011cf" (UID: "47eb3460-dd38-4319-ba98-5ddf0c9011cf"). InnerVolumeSpecName "kube-api-access-9rnkr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:01.534585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.534467 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9rnkr\" (UniqueName: \"kubernetes.io/projected/47eb3460-dd38-4319-ba98-5ddf0c9011cf-kube-api-access-9rnkr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:01.534585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.534512 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:01.534585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:01.534525 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/47eb3460-dd38-4319-ba98-5ddf0c9011cf-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.186344 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.186313 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" Oct 31 00:31:02.186851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.186338 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g" event={"ID":"47eb3460-dd38-4319-ba98-5ddf0c9011cf","Type":"ContainerDied","Data":"ac89cc3003def497c3ea6dd0d2a089a26f3ff2a68612d693585da10d752019dd"} Oct 31 00:31:02.186851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.186414 6234 scope.go:122] "RemoveContainer" containerID="9cc5793cb30d5b34cafa7a3297140a92107fca2c13ddde391c4bfb1015dcc14c" Oct 31 00:31:02.188272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.188246 6234 generic.go:356] "Generic (PLEG): container finished" podID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerID="c0f43c82964f9de5db2181d9315bd2ece5da5128fa2ca196db6df8b7986dbf6d" exitCode=0 Oct 31 00:31:02.188391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.188294 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" event={"ID":"d311da85-ff2b-4b37-aae3-2f268a63b866","Type":"ContainerDied","Data":"c0f43c82964f9de5db2181d9315bd2ece5da5128fa2ca196db6df8b7986dbf6d"} Oct 31 00:31:02.188391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.188338 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" event={"ID":"d311da85-ff2b-4b37-aae3-2f268a63b866","Type":"ContainerDied","Data":"a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7"} Oct 31 00:31:02.188391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.188363 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a456043e4a9e635ba6efddfb030060799523ea9fd695a1110b0052444acc86b7" Oct 31 00:31:02.204440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.204420 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:31:02.218628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.218595 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:31:02.223180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.223153 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w2-14e39b-684d957d89-fs98g"] Oct 31 00:31:02.345639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345594 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345666 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345742 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345866 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345772 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345865 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345913 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.345997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345976 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.346122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.345986 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.346122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346064 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.346243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346125 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\") pod \"d311da85-ff2b-4b37-aae3-2f268a63b866\" (UID: \"d311da85-ff2b-4b37-aae3-2f268a63b866\") " Oct 31 00:31:02.346243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346153 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.346243 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346176 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.346399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346257 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:02.346399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346387 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.346933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346862 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.346933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346888 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d311da85-ff2b-4b37-aae3-2f268a63b866-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.346933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346905 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.346933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346921 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.346933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.346937 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.348183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.348149 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:02.348508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.348486 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.348799 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.348769 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:02.348878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.348829 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt" pod "d311da85-ff2b-4b37-aae3-2f268a63b866" (UID: "d311da85-ff2b-4b37-aae3-2f268a63b866"). InnerVolumeSpecName "kube-api-access-5h8rt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:02.448278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.448240 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5h8rt\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-kube-api-access-5h8rt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.448278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.448273 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.448278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.448282 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:02.448278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:02.448290 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d311da85-ff2b-4b37-aae3-2f268a63b866-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:03.162386 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:03.162350 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:31:03.203690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:03.203660 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv" Oct 31 00:31:03.241822 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:03.241768 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:31:03.248461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:03.248431 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-c60826-openshift-default-6fd5fbb447-8b6kv"] Oct 31 00:31:04.081930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:04.081893 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" path="/var/lib/kubelet/pods/47eb3460-dd38-4319-ba98-5ddf0c9011cf/volumes" Oct 31 00:31:04.082365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:04.082352 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" path="/var/lib/kubelet/pods/d311da85-ff2b-4b37-aae3-2f268a63b866/volumes" Oct 31 00:31:04.105025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:04.104992 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:31:05.454664 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.454535 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:31:05.455218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.454820 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" podUID="0f21f026-cd45-4d72-a915-b1d87334b488" containerName="maas-api" containerID="cri-o://998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21" gracePeriod=30 Oct 31 00:31:05.785932 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.785863 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:31:05.893078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893048 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\") pod \"0f21f026-cd45-4d72-a915-b1d87334b488\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " Oct 31 00:31:05.893265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893217 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\") pod \"0f21f026-cd45-4d72-a915-b1d87334b488\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " Oct 31 00:31:05.893310 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893269 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\") pod \"0f21f026-cd45-4d72-a915-b1d87334b488\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " Oct 31 00:31:05.893385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893363 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\" (UniqueName: \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\") pod \"0f21f026-cd45-4d72-a915-b1d87334b488\" (UID: \"0f21f026-cd45-4d72-a915-b1d87334b488\") " Oct 31 00:31:05.893488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893469 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca" pod "0f21f026-cd45-4d72-a915-b1d87334b488" (UID: "0f21f026-cd45-4d72-a915-b1d87334b488"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:05.893746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.893725 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/0f21f026-cd45-4d72-a915-b1d87334b488-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:05.895525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.895499 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls" pod "0f21f026-cd45-4d72-a915-b1d87334b488" (UID: "0f21f026-cd45-4d72-a915-b1d87334b488"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:05.895618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.895582 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs" pod "0f21f026-cd45-4d72-a915-b1d87334b488" (UID: "0f21f026-cd45-4d72-a915-b1d87334b488"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:05.895659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.895616 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq" pod "0f21f026-cd45-4d72-a915-b1d87334b488" (UID: "0f21f026-cd45-4d72-a915-b1d87334b488"). InnerVolumeSpecName "kube-api-access-t6dqq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:05.995428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.995350 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:05.995428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.995380 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t6dqq\" (UniqueName: \"kubernetes.io/projected/0f21f026-cd45-4d72-a915-b1d87334b488-kube-api-access-t6dqq\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:05.995428 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:05.995392 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/0f21f026-cd45-4d72-a915-b1d87334b488-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.036116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.036076 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" podUID="2baff5db-503a-4da9-875c-3e247765945f" containerName="payload-pre-processing" containerID="cri-o://5951c8a43884faa3efd40312b86bc71a0d70cc10de929729592d869c5c3c61f4" gracePeriod=25 Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.226496 6234 generic.go:356] "Generic (PLEG): container finished" podID="2baff5db-503a-4da9-875c-3e247765945f" containerID="5951c8a43884faa3efd40312b86bc71a0d70cc10de929729592d869c5c3c61f4" exitCode=0 Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.226715 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" event={"ID":"2baff5db-503a-4da9-875c-3e247765945f","Type":"ContainerDied","Data":"5951c8a43884faa3efd40312b86bc71a0d70cc10de929729592d869c5c3c61f4"} Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.228596 6234 generic.go:356] "Generic (PLEG): container finished" podID="0f21f026-cd45-4d72-a915-b1d87334b488" containerID="998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21" exitCode=0 Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.228643 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.228642 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" event={"ID":"0f21f026-cd45-4d72-a915-b1d87334b488","Type":"ContainerDied","Data":"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21"} Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.228810 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp" event={"ID":"0f21f026-cd45-4d72-a915-b1d87334b488","Type":"ContainerDied","Data":"76dd25ab044ee5f4535d2501aea93d48aff0790f401708708ad2608baf23f995"} Oct 31 00:31:06.230925 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.228840 6234 scope.go:122] "RemoveContainer" containerID="998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21" Oct 31 00:31:06.241054 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.241031 6234 scope.go:122] "RemoveContainer" containerID="998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21" Oct 31 00:31:06.241389 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:06.241368 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21\": container with ID starting with 998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21 not found: ID does not exist" containerID="998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21" Oct 31 00:31:06.241483 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.241396 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21"} err="failed to get container status \"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21\": rpc error: code = NotFound desc = could not find container \"998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21\": container with ID starting with 998aabd2ad3c436fb91ce8ec898e982412edf16052aeee8a94ab992efd9d1b21 not found: ID does not exist" Oct 31 00:31:06.253919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.253845 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:31:06.260669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.260641 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w6-0402df-bb7584599-6nfrp"] Oct 31 00:31:06.273130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.273111 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:31:06.301142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.300384 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\") pod \"2baff5db-503a-4da9-875c-3e247765945f\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " Oct 31 00:31:06.301142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.300678 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\" (UniqueName: \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\") pod \"2baff5db-503a-4da9-875c-3e247765945f\" (UID: \"2baff5db-503a-4da9-875c-3e247765945f\") " Oct 31 00:31:06.301142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.300907 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume" pod "2baff5db-503a-4da9-875c-3e247765945f" (UID: "2baff5db-503a-4da9-875c-3e247765945f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:06.301142 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.301096 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2baff5db-503a-4da9-875c-3e247765945f-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.303368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.303339 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw" pod "2baff5db-503a-4da9-875c-3e247765945f" (UID: "2baff5db-503a-4da9-875c-3e247765945f"). InnerVolumeSpecName "kube-api-access-wb4dw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:06.402328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.402293 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wb4dw\" (UniqueName: \"kubernetes.io/projected/2baff5db-503a-4da9-875c-3e247765945f-kube-api-access-wb4dw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:06.958675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.958638 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:31:06.959153 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:06.958875 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" podUID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" containerName="maas-api" containerID="cri-o://bbd9183929df10cd60481b5192eeba326a9b5a0125457cb5d7ef8e24dd88490f" gracePeriod=30 Oct 31 00:31:07.222435 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.222335 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" podUID="2baff5db-503a-4da9-875c-3e247765945f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.112:50052\" within 1s: context deadline exceeded" Oct 31 00:31:07.237448 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:31:07.222912 6234 logging.go:55] [core] [Channel #4512 SubChannel #4513]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.112:50052", ServerName: "10.134.0.112:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.112:50052: operation was canceled" Oct 31 00:31:07.293961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.293926 6234 generic.go:356] "Generic (PLEG): container finished" podID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" containerID="bbd9183929df10cd60481b5192eeba326a9b5a0125457cb5d7ef8e24dd88490f" exitCode=0 Oct 31 00:31:07.294338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.294010 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" event={"ID":"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638","Type":"ContainerDied","Data":"bbd9183929df10cd60481b5192eeba326a9b5a0125457cb5d7ef8e24dd88490f"} Oct 31 00:31:07.294454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.294360 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" event={"ID":"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638","Type":"ContainerDied","Data":"0ccd76327ea3d3f33b8c79f50eb580758aa20ffc61d08341c3e12658c9142d78"} Oct 31 00:31:07.294454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.294389 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0ccd76327ea3d3f33b8c79f50eb580758aa20ffc61d08341c3e12658c9142d78" Oct 31 00:31:07.295702 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.295671 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:31:07.298509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.298476 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" event={"ID":"2baff5db-503a-4da9-875c-3e247765945f","Type":"ContainerDied","Data":"66cbc2af48801c7b644d81fd3bf048fe36c4e54b4a2f4c7689b32cd9a6332bd4"} Oct 31 00:31:07.298668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.298580 6234 scope.go:122] "RemoveContainer" containerID="5951c8a43884faa3efd40312b86bc71a0d70cc10de929729592d869c5c3c61f4" Oct 31 00:31:07.298736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.298716 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx" Oct 31 00:31:07.345992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.345961 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:31:07.360834 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.360802 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-14e39b-76b57cfc95-7ljdx"] Oct 31 00:31:07.425731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.425696 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\") pod \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " Oct 31 00:31:07.425947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.425746 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") pod \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " Oct 31 00:31:07.425947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.425858 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\") pod \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " Oct 31 00:31:07.425947 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.425899 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\" (UniqueName: \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\") pod \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\" (UID: \"a2492e3e-292f-4eeb-93b6-fcf3ac4b4638\") " Oct 31 00:31:07.426134 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.426099 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca" pod "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:07.426345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.426321 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:07.428080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.428059 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls" pod "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:07.428162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.428097 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx" pod "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638"). InnerVolumeSpecName "kube-api-access-j8xvx". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:07.428224 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.428161 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs" pod "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" (UID: "a2492e3e-292f-4eeb-93b6-fcf3ac4b4638"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:07.535526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.535419 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:07.535526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.535463 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j8xvx\" (UniqueName: \"kubernetes.io/projected/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-kube-api-access-j8xvx\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:07.535526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:07.535495 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:08.081084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:08.081049 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f21f026-cd45-4d72-a915-b1d87334b488" path="/var/lib/kubelet/pods/0f21f026-cd45-4d72-a915-b1d87334b488/volumes" Oct 31 00:31:08.081461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:08.081446 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2baff5db-503a-4da9-875c-3e247765945f" path="/var/lib/kubelet/pods/2baff5db-503a-4da9-875c-3e247765945f/volumes" Oct 31 00:31:08.304631 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:08.304601 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt" Oct 31 00:31:08.326016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:08.325987 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:31:08.327937 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:08.327917 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-945abd-6c9f487bfb-98rkt"] Oct 31 00:31:10.081400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:10.081363 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" path="/var/lib/kubelet/pods/a2492e3e-292f-4eeb-93b6-fcf3ac4b4638/volumes" Oct 31 00:31:11.929730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:11.929691 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:31:11.930365 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:11.929982 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" containerID="cri-o://18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da" gracePeriod=30 Oct 31 00:31:12.875877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:12.875832 6234 patch_prober.go:29] interesting pod/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:31:12.876048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:12.875906 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:31:13.736883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.736859 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:31:13.845692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.845599 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\") pod \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " Oct 31 00:31:13.845692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.845663 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\" (UniqueName: \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\") pod \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\" (UID: \"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f\") " Oct 31 00:31:13.846149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.846121 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume" pod "f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" (UID: "f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:13.846302 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.846213 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:13.848295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.848269 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh" pod "f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" (UID: "f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f"). InnerVolumeSpecName "kube-api-access-kljxh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:13.946908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:13.946869 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kljxh\" (UniqueName: \"kubernetes.io/projected/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f-kube-api-access-kljxh\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:14.360734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.360695 6234 generic.go:356] "Generic (PLEG): container finished" podID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerID="9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a" exitCode=0 Oct 31 00:31:14.360976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.360755 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" Oct 31 00:31:14.360976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.360809 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" event={"ID":"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f","Type":"ContainerDied","Data":"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a"} Oct 31 00:31:14.360976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.360866 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6" event={"ID":"f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f","Type":"ContainerDied","Data":"97f68b538cfbdb56a0523b39f2a2698ed9b94f3583da9ec57fa6dc733476af53"} Oct 31 00:31:14.360976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.360889 6234 scope.go:122] "RemoveContainer" containerID="9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a" Oct 31 00:31:14.370794 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.370761 6234 scope.go:122] "RemoveContainer" containerID="9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a" Oct 31 00:31:14.371163 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:14.371143 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a\": container with ID starting with 9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a not found: ID does not exist" containerID="9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a" Oct 31 00:31:14.371237 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.371176 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a"} err="failed to get container status \"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a\": rpc error: code = NotFound desc = could not find container \"9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a\": container with ID starting with 9274e0c95f9a1a94d59363b481ff88c4a0e3f66a4c708127b193f3d2d5699e8a not found: ID does not exist" Oct 31 00:31:14.382265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.382217 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:31:14.385507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:14.385483 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w6-0402df-57775579dc-mjnm6"] Oct 31 00:31:15.150536 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:15.150499 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:31:15.151057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:15.150755 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" containerID="cri-o://16f8100ac391b1af1f4a85accb81c423d998483561f57b3938612fda336fd21b" gracePeriod=30 Oct 31 00:31:15.160672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:15.160643 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:31:15.160935 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:15.160906 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" containerID="cri-o://67f3c104a63781011eeadebc8d2092605d535082e205077fba26e39a5e2f3d93" gracePeriod=30 Oct 31 00:31:16.081081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:16.081042 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" path="/var/lib/kubelet/pods/f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f/volumes" Oct 31 00:31:17.197600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.197563 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:31:17.294323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294205 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294250 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294301 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294390 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294447 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294534 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294659 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294634 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294655 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.294944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294669 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294773 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\") pod \"01b77f4c-be6f-4bec-ad2c-90f342187369\" (UID: \"01b77f4c-be6f-4bec-ad2c-90f342187369\") " Oct 31 00:31:17.294944 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.294920 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.295161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295024 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.295429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295404 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.295577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295296 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:17.295633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295579 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.295633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295602 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.295633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295616 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.295633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.295631 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.297050 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.297024 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:17.298033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.298006 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "kube-api-access-x99xw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:17.298149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.298035 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:17.298149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.298050 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy" pod "01b77f4c-be6f-4bec-ad2c-90f342187369" (UID: "01b77f4c-be6f-4bec-ad2c-90f342187369"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:17.381229 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.381189 6234 generic.go:356] "Generic (PLEG): container finished" podID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerID="18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da" exitCode=0 Oct 31 00:31:17.381414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.381255 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" Oct 31 00:31:17.381414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.381270 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerDied","Data":"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da"} Oct 31 00:31:17.381414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.381346 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd" event={"ID":"01b77f4c-be6f-4bec-ad2c-90f342187369","Type":"ContainerDied","Data":"4388d6952b3e97ff1b4e037f21254051a984997e3b18229b93030f95ec88605f"} Oct 31 00:31:17.381414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.381369 6234 scope.go:122] "RemoveContainer" containerID="18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da" Oct 31 00:31:17.392707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.392686 6234 scope.go:122] "RemoveContainer" containerID="24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383" Oct 31 00:31:17.396363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.396335 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.396363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.396362 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x99xw\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-kube-api-access-x99xw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.396543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.396375 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/01b77f4c-be6f-4bec-ad2c-90f342187369-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.396543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.396384 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.396543 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.396392 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/01b77f4c-be6f-4bec-ad2c-90f342187369-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:17.401768 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.401746 6234 scope.go:122] "RemoveContainer" containerID="18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da" Oct 31 00:31:17.402089 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:17.402070 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da\": container with ID starting with 18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da not found: ID does not exist" containerID="18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da" Oct 31 00:31:17.402144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.402102 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da"} err="failed to get container status \"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da\": rpc error: code = NotFound desc = could not find container \"18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da\": container with ID starting with 18778868e9b9db7e827a2ea59430c4ad6033939aa4f3aeb3de7734b3e05de4da not found: ID does not exist" Oct 31 00:31:17.402144 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.402119 6234 scope.go:122] "RemoveContainer" containerID="24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383" Oct 31 00:31:17.402351 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:17.402336 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383\": container with ID starting with 24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383 not found: ID does not exist" containerID="24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383" Oct 31 00:31:17.402389 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.402355 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383"} err="failed to get container status \"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383\": rpc error: code = NotFound desc = could not find container \"24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383\": container with ID starting with 24ac7eed36ed8dede112daf3665d5582a3bacd10f4938f08eee3384e07c94383 not found: ID does not exist" Oct 31 00:31:17.408390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.408351 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:31:17.410978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.410957 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-14e39b-openshift-default-56d8c66565-dwnqd"] Oct 31 00:31:17.453355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.453322 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:31:17.473116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:17.473085 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:31:18.081045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:18.081018 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" path="/var/lib/kubelet/pods/01b77f4c-be6f-4bec-ad2c-90f342187369/volumes" Oct 31 00:31:20.404910 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.404878 6234 generic.go:356] "Generic (PLEG): container finished" podID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerID="16f8100ac391b1af1f4a85accb81c423d998483561f57b3938612fda336fd21b" exitCode=0 Oct 31 00:31:20.405298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.404942 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerDied","Data":"16f8100ac391b1af1f4a85accb81c423d998483561f57b3938612fda336fd21b"} Oct 31 00:31:20.405298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.405010 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" event={"ID":"e268a382-a45c-48c9-adc5-cb3c7b117920","Type":"ContainerDied","Data":"9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9"} Oct 31 00:31:20.405298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.405038 6234 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f901282ec9a51a13979649796a230acad21ffb5b615baccc0606bee36fd20d9" Oct 31 00:31:20.405298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.405055 6234 scope.go:122] "RemoveContainer" containerID="6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02" Oct 31 00:31:20.407086 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.407056 6234 generic.go:356] "Generic (PLEG): container finished" podID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerID="67f3c104a63781011eeadebc8d2092605d535082e205077fba26e39a5e2f3d93" exitCode=0 Oct 31 00:31:20.407205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.407100 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" event={"ID":"fed08b4a-3f3f-49e8-a91f-0e70ebffd914","Type":"ContainerDied","Data":"67f3c104a63781011eeadebc8d2092605d535082e205077fba26e39a5e2f3d93"} Oct 31 00:31:20.418985 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:20.418913 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02\": container with ID starting with 6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02 not found: ID does not exist" containerID="6f967391173da8e6fd4726df5771b0095141bd5df685ac25e129c6b3f352dc02" Oct 31 00:31:20.422469 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.422450 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:31:20.434960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.434940 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.434991 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435023 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435068 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435091 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435136 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435194 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435226 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.435325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435230 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435407 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\") pod \"e268a382-a45c-48c9-adc5-cb3c7b117920\" (UID: \"e268a382-a45c-48c9-adc5-cb3c7b117920\") " Oct 31 00:31:20.435532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435241 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.435532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435455 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.435697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435600 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.435998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.435842 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:20.436171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.436098 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.436242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.436176 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.436242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.436192 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.436242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.436206 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.436242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.436221 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e268a382-a45c-48c9-adc5-cb3c7b117920-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.437646 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.437624 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:20.437730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.437665 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.437730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.437681 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:20.438022 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.438002 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw" pod "e268a382-a45c-48c9-adc5-cb3c7b117920" (UID: "e268a382-a45c-48c9-adc5-cb3c7b117920"). InnerVolumeSpecName "kube-api-access-krxlw". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:20.458597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.458563 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:31:20.536769 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.536737 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.536943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.536775 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.536943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.536836 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.536943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.536894 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.536943 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.536934 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.537178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537025 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.537178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537067 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.537178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537088 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.537178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537132 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.537178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537162 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\") pod \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\" (UID: \"fed08b4a-3f3f-49e8-a91f-0e70ebffd914\") " Oct 31 00:31:20.537439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.537439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537338 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.537439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537373 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.537759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537574 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:20.537759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537699 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-krxlw\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-kube-api-access-krxlw\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.537759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537719 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.537759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537733 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.537759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537746 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.538071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537760 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.538071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537774 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e268a382-a45c-48c9-adc5-cb3c7b117920-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.538071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537805 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.538071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537819 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.538071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.537831 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.539275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.539249 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:20.539532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.539510 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:20.539670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.539650 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "kube-api-access-89k48". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:20.539862 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.539838 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token" pod "fed08b4a-3f3f-49e8-a91f-0e70ebffd914" (UID: "fed08b4a-3f3f-49e8-a91f-0e70ebffd914"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:20.639214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.639170 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.639214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.639207 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-89k48\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-kube-api-access-89k48\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.639214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.639220 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:20.639460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:20.639229 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fed08b4a-3f3f-49e8-a91f-0e70ebffd914-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:21.414544 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.414509 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" Oct 31 00:31:21.415063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.414539 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm" event={"ID":"fed08b4a-3f3f-49e8-a91f-0e70ebffd914","Type":"ContainerDied","Data":"ced68ad8c85358a0a19e4d33e0ceb7ad59786926a066920db22a09b8c8c0b181"} Oct 31 00:31:21.415063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.414600 6234 scope.go:122] "RemoveContainer" containerID="67f3c104a63781011eeadebc8d2092605d535082e205077fba26e39a5e2f3d93" Oct 31 00:31:21.416377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.416357 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf" Oct 31 00:31:21.447975 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.447950 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:31:21.455161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.455141 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-wm2xm"] Oct 31 00:31:21.472240 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.472211 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:31:21.478356 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:21.478332 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-0402df-openshift-default-8665fdc84f-n9kpf"] Oct 31 00:31:22.082603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:22.082566 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" path="/var/lib/kubelet/pods/e268a382-a45c-48c9-adc5-cb3c7b117920/volumes" Oct 31 00:31:22.083131 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:22.083109 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" path="/var/lib/kubelet/pods/fed08b4a-3f3f-49e8-a91f-0e70ebffd914/volumes" Oct 31 00:31:22.924565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:22.924529 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:31:27.925569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:27.925481 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" containerID="cri-o://56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456" gracePeriod=25 Oct 31 00:31:31.083200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:31.083160 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:31:32.474884 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.474842 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" containerID="cri-o://9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb" gracePeriod=45 Oct 31 00:31:32.497265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.497211 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" containerID="cri-o://20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c" gracePeriod=45 Oct 31 00:31:32.525610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.525577 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" probeResult="failure" output="Get \"https://10.134.0.59:8000/ready\": dial tcp 10.134.0.59:8000: connect: connection refused" Oct 31 00:31:32.782592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.782544 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:31:32.859400 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.859374 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:31:32.939419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939373 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.939622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939430 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.939622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939462 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\" (UniqueName: \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.939622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939601 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\" (UniqueName: \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.939829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939651 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.939829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939695 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.939829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939700 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.939829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939754 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.939829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939815 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.940117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939846 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\" (UniqueName: \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.940117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939892 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.940117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939932 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\" (UniqueName: \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.940117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939965 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\") pod \"b1516214-0b59-4ab0-ae3e-eb4aad219009\" (UID: \"b1516214-0b59-4ab0-ae3e-eb4aad219009\") " Oct 31 00:31:32.940117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.939996 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\") pod \"77222f3c-662b-4453-aa15-b2be294e2526\" (UID: \"77222f3c-662b-4453-aa15-b2be294e2526\") " Oct 31 00:31:32.940309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940184 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.940417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940394 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.940876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940628 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:32.940876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940652 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:32.940876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940667 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:32.940876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.940736 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.941655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.941520 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.943617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.943443 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.943617 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.943473 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "kube-api-access-v6866". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:32.943736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.943648 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:32.944155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.944135 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "kube-api-access-2zhkp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:32.944309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.944286 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs" pod "77222f3c-662b-4453-aa15-b2be294e2526" (UID: "77222f3c-662b-4453-aa15-b2be294e2526"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:32.944580 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:32.944552 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs" pod "b1516214-0b59-4ab0-ae3e-eb4aad219009" (UID: "b1516214-0b59-4ab0-ae3e-eb4aad219009"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041737 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77222f3c-662b-4453-aa15-b2be294e2526-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041767 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041795 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041809 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b1516214-0b59-4ab0-ae3e-eb4aad219009-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041821 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041833 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v6866\" (UniqueName: \"kubernetes.io/projected/b1516214-0b59-4ab0-ae3e-eb4aad219009-kube-api-access-v6866\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041847 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/b1516214-0b59-4ab0-ae3e-eb4aad219009-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.041868 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041869 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77222f3c-662b-4453-aa15-b2be294e2526-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.042242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.041883 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2zhkp\" (UniqueName: \"kubernetes.io/projected/77222f3c-662b-4453-aa15-b2be294e2526-kube-api-access-2zhkp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:33.169264 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:33.169226 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:31:33.512340 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.512302 6234 generic.go:356] "Generic (PLEG): container finished" podID="77222f3c-662b-4453-aa15-b2be294e2526" containerID="9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb" exitCode=0 Oct 31 00:31:33.512821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.512353 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" Oct 31 00:31:33.512821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.512395 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" event={"ID":"77222f3c-662b-4453-aa15-b2be294e2526","Type":"ContainerDied","Data":"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb"} Oct 31 00:31:33.512821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.512454 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq" event={"ID":"77222f3c-662b-4453-aa15-b2be294e2526","Type":"ContainerDied","Data":"4d435af8b762c0e2f27dc871d54105c123e6db86e90636ce0c04e18c9c298aa7"} Oct 31 00:31:33.512821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.512481 6234 scope.go:122] "RemoveContainer" containerID="9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb" Oct 31 00:31:33.513935 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.513914 6234 generic.go:356] "Generic (PLEG): container finished" podID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerID="20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c" exitCode=0 Oct 31 00:31:33.514008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.513958 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" event={"ID":"b1516214-0b59-4ab0-ae3e-eb4aad219009","Type":"ContainerDied","Data":"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c"} Oct 31 00:31:33.514008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.513987 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" event={"ID":"b1516214-0b59-4ab0-ae3e-eb4aad219009","Type":"ContainerDied","Data":"b31495dc7567f12b109c203945c82b070c1eca25788d25b1ce53885252027675"} Oct 31 00:31:33.514193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.514175 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx" Oct 31 00:31:33.523331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.523314 6234 scope.go:122] "RemoveContainer" containerID="9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb" Oct 31 00:31:33.523556 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:33.523534 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb\": container with ID starting with 9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb not found: ID does not exist" containerID="9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb" Oct 31 00:31:33.523645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.523562 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb"} err="failed to get container status \"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb\": rpc error: code = NotFound desc = could not find container \"9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb\": container with ID starting with 9b38d37f62493350834cc6bee036068eaf3287c7ac40a5e0d04afa130ab5a7eb not found: ID does not exist" Oct 31 00:31:33.523645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.523577 6234 scope.go:122] "RemoveContainer" containerID="20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c" Oct 31 00:31:33.532470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.532452 6234 scope.go:122] "RemoveContainer" containerID="20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c" Oct 31 00:31:33.532731 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:33.532713 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c\": container with ID starting with 20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c not found: ID does not exist" containerID="20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c" Oct 31 00:31:33.532797 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.532738 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c"} err="failed to get container status \"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c\": rpc error: code = NotFound desc = could not find container \"20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c\": container with ID starting with 20b8dfb4d27e6ac6a8f4bd2d38a75f2213c2224a7dd5aa3e8b21afb0474a884c not found: ID does not exist" Oct 31 00:31:33.553127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.553089 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:31:33.561384 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.561353 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/premium-w2-7c5da3-kserve-5bbb77fb6f-njksx"] Oct 31 00:31:33.582265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.582226 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:31:33.601687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:33.601656 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w2-7c5da3/distinct-2-w2-7c5da3-kserve-746d88b67f-pn7kq"] Oct 31 00:31:34.081193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:34.081157 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="77222f3c-662b-4453-aa15-b2be294e2526" path="/var/lib/kubelet/pods/77222f3c-662b-4453-aa15-b2be294e2526/volumes" Oct 31 00:31:34.081711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:34.081689 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" path="/var/lib/kubelet/pods/b1516214-0b59-4ab0-ae3e-eb4aad219009/volumes" Oct 31 00:31:41.083286 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:41.083248 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:31:42.537037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:42.537006 6234 scope.go:122] "RemoveContainer" containerID="c0f43c82964f9de5db2181d9315bd2ece5da5128fa2ca196db6df8b7986dbf6d" Oct 31 00:31:42.546321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:42.546301 6234 scope.go:122] "RemoveContainer" containerID="bbd9183929df10cd60481b5192eeba326a9b5a0125457cb5d7ef8e24dd88490f" Oct 31 00:31:42.564256 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:42.564219 6234 scope.go:122] "RemoveContainer" containerID="16f8100ac391b1af1f4a85accb81c423d998483561f57b3938612fda336fd21b" Oct 31 00:31:48.099033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.098988 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:31:48.099440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.099181 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" containerID="cri-o://bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951" gracePeriod=30 Oct 31 00:31:48.248449 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.248421 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:31:48.324113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.324080 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\" (UniqueName: \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\") pod \"6f0be845-0d78-4855-99e8-fceee5d1db1e\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " Oct 31 00:31:48.324113 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.324117 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\") pod \"6f0be845-0d78-4855-99e8-fceee5d1db1e\" (UID: \"6f0be845-0d78-4855-99e8-fceee5d1db1e\") " Oct 31 00:31:48.324575 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.324555 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume" pod "6f0be845-0d78-4855-99e8-fceee5d1db1e" (UID: "6f0be845-0d78-4855-99e8-fceee5d1db1e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:48.326341 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.326309 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5" pod "6f0be845-0d78-4855-99e8-fceee5d1db1e" (UID: "6f0be845-0d78-4855-99e8-fceee5d1db1e"). InnerVolumeSpecName "kube-api-access-x84x5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:48.425149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.425112 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x84x5\" (UniqueName: \"kubernetes.io/projected/6f0be845-0d78-4855-99e8-fceee5d1db1e-kube-api-access-x84x5\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:48.425149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.425142 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f0be845-0d78-4855-99e8-fceee5d1db1e-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:48.608305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.608222 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:31:48.632442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.632410 6234 generic.go:356] "Generic (PLEG): container finished" podID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerID="56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456" exitCode=0 Oct 31 00:31:48.632639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.632461 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" Oct 31 00:31:48.632639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.632493 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" event={"ID":"6f0be845-0d78-4855-99e8-fceee5d1db1e","Type":"ContainerDied","Data":"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456"} Oct 31 00:31:48.632639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.632541 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4" event={"ID":"6f0be845-0d78-4855-99e8-fceee5d1db1e","Type":"ContainerDied","Data":"7faf50e13602c647f4e58386d8414da138f6bceb1a026ce3bfb7a0cff5ebdc47"} Oct 31 00:31:48.632639 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.632563 6234 scope.go:122] "RemoveContainer" containerID="56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456" Oct 31 00:31:48.641537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.641512 6234 scope.go:122] "RemoveContainer" containerID="56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456" Oct 31 00:31:48.641908 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:48.641887 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456\": container with ID starting with 56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456 not found: ID does not exist" containerID="56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456" Oct 31 00:31:48.641978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.641919 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456"} err="failed to get container status \"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456\": rpc error: code = NotFound desc = could not find container \"56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456\": container with ID starting with 56a1e2953f9c1a413d8711c8083958dc1b050d749987ec53899bc373c7faf456 not found: ID does not exist" Oct 31 00:31:48.670125 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.670081 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:31:48.790321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.790284 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:31:48.832073 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:48.832044 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-7c5da3-7b84bc6645-cmmb4"] Oct 31 00:31:50.079998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:50.079962 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" path="/var/lib/kubelet/pods/6f0be845-0d78-4855-99e8-fceee5d1db1e/volumes" Oct 31 00:31:53.349497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.349470 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:31:53.482079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482039 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482083 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482159 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482183 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482212 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482239 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482316 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482347 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482358 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.482613 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482466 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.482776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482640 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.482776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482668 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.482912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482796 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\") pod \"801ed45b-760b-456a-ad10-178de4ed72e4\" (UID: \"801ed45b-760b-456a-ad10-178de4ed72e4\") " Oct 31 00:31:53.482964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.482935 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:31:53.483427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.483395 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.483427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.483421 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.483622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.483437 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.483622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.483452 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.483622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.483467 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/801ed45b-760b-456a-ad10-178de4ed72e4-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.484683 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.484655 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:31:53.484810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.484757 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.484957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.484936 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:31:53.485223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.485204 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm" pod "801ed45b-760b-456a-ad10-178de4ed72e4" (UID: "801ed45b-760b-456a-ad10-178de4ed72e4"). InnerVolumeSpecName "kube-api-access-pckhm". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:31:53.584337 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.584297 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pckhm\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-kube-api-access-pckhm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.584337 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.584332 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/801ed45b-760b-456a-ad10-178de4ed72e4-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.584337 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.584342 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/801ed45b-760b-456a-ad10-178de4ed72e4-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.584572 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.584351 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/801ed45b-760b-456a-ad10-178de4ed72e4-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:31:53.608911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.608881 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" containerID="cri-o://71c7c409d8af9b2040339853ecd07605c77541f3626418fc8510c609dd5af533" gracePeriod=25 Oct 31 00:31:53.660178 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.660148 6234 generic.go:356] "Generic (PLEG): container finished" podID="801ed45b-760b-456a-ad10-178de4ed72e4" containerID="bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951" exitCode=0 Oct 31 00:31:53.660319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.660193 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" Oct 31 00:31:53.660319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.660241 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" event={"ID":"801ed45b-760b-456a-ad10-178de4ed72e4","Type":"ContainerDied","Data":"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951"} Oct 31 00:31:53.660319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.660295 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4" event={"ID":"801ed45b-760b-456a-ad10-178de4ed72e4","Type":"ContainerDied","Data":"897752837e3075c6390d6fafee08b57502840c880498d3273eb33ce2486f6a87"} Oct 31 00:31:53.660319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.660317 6234 scope.go:122] "RemoveContainer" containerID="bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951" Oct 31 00:31:53.670268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.670248 6234 scope.go:122] "RemoveContainer" containerID="bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951" Oct 31 00:31:53.670339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.670313 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" containerID="cri-o://48bfe8bdb01b104a477e99969892f3e89f82a019a6b047cabcaf5af1963336dc" gracePeriod=25 Oct 31 00:31:53.670606 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:31:53.670564 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951\": container with ID starting with bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951 not found: ID does not exist" containerID="bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951" Oct 31 00:31:53.670606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.670593 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951"} err="failed to get container status \"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951\": rpc error: code = NotFound desc = could not find container \"bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951\": container with ID starting with bf558d7fe145d3d566b0c1f18f399497799c53e1f277b9bd4f1ec9b626ba4951 not found: ID does not exist" Oct 31 00:31:53.685171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.685141 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:31:53.689056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:53.689035 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-7c5da3-openshift-default-6fcfbf7d47-2plj4"] Oct 31 00:31:54.080929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:31:54.080893 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" path="/var/lib/kubelet/pods/801ed45b-760b-456a-ad10-178de4ed72e4/volumes" Oct 31 00:32:00.053609 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:00.053563 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:32:00.054109 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:00.053567 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:32:03.160820 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:32:03.160765 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:32:03.964532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:03.964486 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:32:03.964904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:03.964839 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" podUID="2fc57b11-8551-469c-92b3-611656e2615c" containerName="maas-api" containerID="cri-o://508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f" gracePeriod=30 Oct 31 00:32:04.227695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.227611 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:32:04.321095 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.321054 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\") pod \"2fc57b11-8551-469c-92b3-611656e2615c\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " Oct 31 00:32:04.321281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.321142 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\" (UniqueName: \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\") pod \"2fc57b11-8551-469c-92b3-611656e2615c\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " Oct 31 00:32:04.321281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.321236 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") pod \"2fc57b11-8551-469c-92b3-611656e2615c\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " Oct 31 00:32:04.321281 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.321266 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\") pod \"2fc57b11-8551-469c-92b3-611656e2615c\" (UID: \"2fc57b11-8551-469c-92b3-611656e2615c\") " Oct 31 00:32:04.321716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.321684 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca" pod "2fc57b11-8551-469c-92b3-611656e2615c" (UID: "2fc57b11-8551-469c-92b3-611656e2615c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:04.323340 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.323318 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs" pod "2fc57b11-8551-469c-92b3-611656e2615c" (UID: "2fc57b11-8551-469c-92b3-611656e2615c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:04.323440 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.323378 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk" pod "2fc57b11-8551-469c-92b3-611656e2615c" (UID: "2fc57b11-8551-469c-92b3-611656e2615c"). InnerVolumeSpecName "kube-api-access-r5mmk". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:04.323484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.323441 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls" pod "2fc57b11-8551-469c-92b3-611656e2615c" (UID: "2fc57b11-8551-469c-92b3-611656e2615c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:04.423806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.423079 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:04.423806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.423115 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r5mmk\" (UniqueName: \"kubernetes.io/projected/2fc57b11-8551-469c-92b3-611656e2615c-kube-api-access-r5mmk\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:04.423806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.423133 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/2fc57b11-8551-469c-92b3-611656e2615c-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:04.423806 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.423149 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/2fc57b11-8551-469c-92b3-611656e2615c-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:04.724771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.724736 6234 generic.go:356] "Generic (PLEG): container finished" podID="2fc57b11-8551-469c-92b3-611656e2615c" containerID="508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f" exitCode=0 Oct 31 00:32:04.724995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.724823 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" Oct 31 00:32:04.724995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.724819 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" event={"ID":"2fc57b11-8551-469c-92b3-611656e2615c","Type":"ContainerDied","Data":"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f"} Oct 31 00:32:04.724995 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.724965 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr" event={"ID":"2fc57b11-8551-469c-92b3-611656e2615c","Type":"ContainerDied","Data":"3b233f95ab1ad1031ad8c6f96b4f773670018016018257a0be0f327e6731f9ac"} Oct 31 00:32:04.725181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.724996 6234 scope.go:122] "RemoveContainer" containerID="508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f" Oct 31 00:32:04.734588 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.734567 6234 scope.go:122] "RemoveContainer" containerID="508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f" Oct 31 00:32:04.734955 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:32:04.734932 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f\": container with ID starting with 508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f not found: ID does not exist" containerID="508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f" Oct 31 00:32:04.735060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.734961 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f"} err="failed to get container status \"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f\": rpc error: code = NotFound desc = could not find container \"508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f\": container with ID starting with 508e6b8ef45dcd85609c9784877bcecd0046625e03f44d841a15375a6815e22f not found: ID does not exist" Oct 31 00:32:04.774108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.774064 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:32:04.792823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:04.792769 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w5-7b2341-58ff487bb9-rx2rr"] Oct 31 00:32:06.080136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:06.080093 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2fc57b11-8551-469c-92b3-611656e2615c" path="/var/lib/kubelet/pods/2fc57b11-8551-469c-92b3-611656e2615c/volumes" Oct 31 00:32:10.052626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:10.052581 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:32:10.053060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:10.052593 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:32:11.900080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:11.900047 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:32:11.901052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:11.900330 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" containerID="cri-o://c1ecfe013134fa8eab807008b4d3d85f866e7dc2c53609c274be2869f623465d" gracePeriod=30 Oct 31 00:32:11.902368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:11.902340 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:32:11.902540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:11.902518 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" containerID="cri-o://a59651bf71f42f4ef4e0b0c90a7266522edc271235de1c07584edda77efeff0e" gracePeriod=30 Oct 31 00:32:12.081851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:12.081821 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:32:12.082044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:12.081876 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:13.795249 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.795192 6234 generic.go:356] "Generic (PLEG): container finished" podID="a29a19b4-12de-464e-8b28-9f1598267fea" containerID="71c7c409d8af9b2040339853ecd07605c77541f3626418fc8510c609dd5af533" exitCode=0 Oct 31 00:32:13.795719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.795273 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" event={"ID":"a29a19b4-12de-464e-8b28-9f1598267fea","Type":"ContainerDied","Data":"71c7c409d8af9b2040339853ecd07605c77541f3626418fc8510c609dd5af533"} Oct 31 00:32:13.797266 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.797239 6234 generic.go:356] "Generic (PLEG): container finished" podID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerID="48bfe8bdb01b104a477e99969892f3e89f82a019a6b047cabcaf5af1963336dc" exitCode=0 Oct 31 00:32:13.797422 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.797314 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" event={"ID":"2467d0c4-552c-4916-9b15-ac1aaea4c5c1","Type":"ContainerDied","Data":"48bfe8bdb01b104a477e99969892f3e89f82a019a6b047cabcaf5af1963336dc"} Oct 31 00:32:13.830654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.830614 6234 patch_prober.go:29] interesting pod/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:32:13.830867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.830683 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:32:13.925499 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.925468 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:32:13.943911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:13.943889 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:32:14.042130 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042092 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\" (UniqueName: \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\") pod \"a29a19b4-12de-464e-8b28-9f1598267fea\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " Oct 31 00:32:14.042309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042146 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\" (UniqueName: \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\") pod \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " Oct 31 00:32:14.042309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042185 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\") pod \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\" (UID: \"2467d0c4-552c-4916-9b15-ac1aaea4c5c1\") " Oct 31 00:32:14.042309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042214 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\") pod \"a29a19b4-12de-464e-8b28-9f1598267fea\" (UID: \"a29a19b4-12de-464e-8b28-9f1598267fea\") " Oct 31 00:32:14.042611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042584 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume" pod "2467d0c4-552c-4916-9b15-ac1aaea4c5c1" (UID: "2467d0c4-552c-4916-9b15-ac1aaea4c5c1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:14.042741 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.042619 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume" pod "a29a19b4-12de-464e-8b28-9f1598267fea" (UID: "a29a19b4-12de-464e-8b28-9f1598267fea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:14.044419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.044394 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb" pod "a29a19b4-12de-464e-8b28-9f1598267fea" (UID: "a29a19b4-12de-464e-8b28-9f1598267fea"). InnerVolumeSpecName "kube-api-access-9b6nb". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:14.044496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.044417 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7" pod "2467d0c4-552c-4916-9b15-ac1aaea4c5c1" (UID: "2467d0c4-552c-4916-9b15-ac1aaea4c5c1"). InnerVolumeSpecName "kube-api-access-vhjq7". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:14.143590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.143508 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vhjq7\" (UniqueName: \"kubernetes.io/projected/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-kube-api-access-vhjq7\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.143590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.143544 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2467d0c4-552c-4916-9b15-ac1aaea4c5c1-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.143590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.143557 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a29a19b4-12de-464e-8b28-9f1598267fea-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.143590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.143571 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9b6nb\" (UniqueName: \"kubernetes.io/projected/a29a19b4-12de-464e-8b28-9f1598267fea-kube-api-access-9b6nb\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:14.803501 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.803473 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" Oct 31 00:32:14.804028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.803510 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd" event={"ID":"2467d0c4-552c-4916-9b15-ac1aaea4c5c1","Type":"ContainerDied","Data":"f2d1fe77d98e421f42f5f368f0d77b55f3f4f914396686786268af8e2d4eaf52"} Oct 31 00:32:14.804028 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.803579 6234 scope.go:122] "RemoveContainer" containerID="48bfe8bdb01b104a477e99969892f3e89f82a019a6b047cabcaf5af1963336dc" Oct 31 00:32:14.804982 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.804953 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" Oct 31 00:32:14.805048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.804958 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d" event={"ID":"a29a19b4-12de-464e-8b28-9f1598267fea","Type":"ContainerDied","Data":"28980a58f7cfc2ba3f48154941fb1626ac511d71c3f41f33411ea7be3394ce93"} Oct 31 00:32:14.815466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.815444 6234 scope.go:122] "RemoveContainer" containerID="71c7c409d8af9b2040339853ecd07605c77541f3626418fc8510c609dd5af533" Oct 31 00:32:14.829652 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.829625 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:32:14.833343 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.833321 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w5-7b2341-6468465ddc-sjl4d"] Oct 31 00:32:14.846063 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.846039 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:32:14.854385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:14.854363 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w5-7b2341-786ff4c547-l4wzd"] Oct 31 00:32:16.084776 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.084735 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" path="/var/lib/kubelet/pods/2467d0c4-552c-4916-9b15-ac1aaea4c5c1/volumes" Oct 31 00:32:16.085360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.085336 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" path="/var/lib/kubelet/pods/a29a19b4-12de-464e-8b28-9f1598267fea/volumes" Oct 31 00:32:16.306419 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.306381 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:16.307055 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307032 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307056 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307074 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307082 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307093 6234 cpu_manager.go:414] "containerMap: removing container" podUID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" containerName="payload-processing" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307104 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" containerName="payload-processing" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307122 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307130 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307141 6234 cpu_manager.go:414] "containerMap: removing container" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307149 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307161 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307168 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307173 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307178 6234 cpu_manager.go:414] "containerMap: removing container" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307185 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307201 6234 cpu_manager.go:414] "containerMap: removing container" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307210 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307222 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2baff5db-503a-4da9-875c-3e247765945f" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307230 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2baff5db-503a-4da9-875c-3e247765945f" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307240 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307248 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307260 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307267 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307276 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307283 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307296 6234 cpu_manager.go:414] "containerMap: removing container" podUID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307303 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307315 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307321 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307329 6234 cpu_manager.go:414] "containerMap: removing container" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307336 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307346 6234 cpu_manager.go:414] "containerMap: removing container" podUID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307353 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307363 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307370 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307383 6234 cpu_manager.go:414] "containerMap: removing container" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307390 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307402 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307410 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307424 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307431 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307442 6234 cpu_manager.go:414] "containerMap: removing container" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307451 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307461 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307468 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307480 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307487 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307502 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307510 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307522 6234 cpu_manager.go:414] "containerMap: removing container" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307529 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307539 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307546 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307557 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307564 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307576 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307583 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307593 6234 cpu_manager.go:414] "containerMap: removing container" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307602 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307612 6234 cpu_manager.go:414] "containerMap: removing container" podUID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307619 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307629 6234 cpu_manager.go:414] "containerMap: removing container" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307636 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307646 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307653 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307668 6234 cpu_manager.go:414] "containerMap: removing container" podUID="0f21f026-cd45-4d72-a915-b1d87334b488" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307676 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f21f026-cd45-4d72-a915-b1d87334b488" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307687 6234 cpu_manager.go:414] "containerMap: removing container" podUID="73170cf3-9711-40a6-95a2-5e013a98bc12" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307695 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73170cf3-9711-40a6-95a2-5e013a98bc12" containerName="payload-pre-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307711 6234 cpu_manager.go:414] "containerMap: removing container" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307718 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307732 6234 cpu_manager.go:414] "containerMap: removing container" podUID="2fc57b11-8551-469c-92b3-611656e2615c" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307739 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2fc57b11-8551-469c-92b3-611656e2615c" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307750 6234 cpu_manager.go:414] "containerMap: removing container" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307757 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307771 6234 cpu_manager.go:414] "containerMap: removing container" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307799 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307811 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307817 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307828 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307835 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" containerName="maas-api" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307848 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307857 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307869 6234 cpu_manager.go:414] "containerMap: removing container" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307877 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307891 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307898 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.307859 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307906 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307913 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307925 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307932 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307942 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307949 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307958 6234 cpu_manager.go:414] "containerMap: removing container" podUID="f8eda208-4692-45bd-ba91-9eb470e5104c" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307966 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f8eda208-4692-45bd-ba91-9eb470e5104c" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307980 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307988 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.307997 6234 cpu_manager.go:414] "containerMap: removing container" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308004 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308108 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f169e7e9-2bd0-4b9c-bd7a-ce62509ac47f" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308123 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="fed08b4a-3f3f-49e8-a91f-0e70ebffd914" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308135 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308143 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f1f38638-649c-42ae-afac-6eb04bfbdd91" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308153 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308161 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="801ed45b-760b-456a-ad10-178de4ed72e4" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308169 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="374240e2-0c6a-4ec6-9db8-b07d279a49ad" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308178 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308186 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308195 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f21f026-cd45-4d72-a915-b1d87334b488" containerName="maas-api" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308205 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="73170cf3-9711-40a6-95a2-5e013a98bc12" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308214 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e268a382-a45c-48c9-adc5-cb3c7b117920" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308225 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2fc57b11-8551-469c-92b3-611656e2615c" containerName="maas-api" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308235 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="ad4bbafa-37e8-403d-84a6-6c4437086688" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308245 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308253 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="75668942-ad30-4251-ad4b-b5df786ff9ae" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308264 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308273 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a2492e3e-292f-4eeb-93b6-fcf3ac4b4638" containerName="maas-api" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308282 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a29a19b4-12de-464e-8b28-9f1598267fea" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308291 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308299 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308307 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6d115f3-0494-4a5d-bf51-d3358c96c1b9" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308317 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308326 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b993a36b-71fe-4361-a86d-fd8e6b4bd396" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308336 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8e2ec100-4d8c-4cce-974c-18ab8b60f3cc" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308346 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="f8eda208-4692-45bd-ba91-9eb470e5104c" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308354 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c13ba68-b7ce-40f4-8232-c8d2203cd6b4" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308363 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308371 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6a93d93-30dc-43fc-a3a3-b640229dfc1f" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308381 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308389 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="639d6597-1d61-4757-8dfe-2e4ddf1e57f3" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308397 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2467d0c4-552c-4916-9b15-ac1aaea4c5c1" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308406 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="0d148c2b-1b9f-4782-acbe-42c5b821a395" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308416 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308425 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="2baff5db-503a-4da9-875c-3e247765945f" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308434 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308442 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308450 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d311da85-ff2b-4b37-aae3-2f268a63b866" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308458 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b85dda1-ff52-42e7-b9fd-f54f6fcb803f" containerName="maas-api" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308467 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="b1516214-0b59-4ab0-ae3e-eb4aad219009" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308476 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="77222f3c-662b-4453-aa15-b2be294e2526" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308485 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="47eb3460-dd38-4319-ba98-5ddf0c9011cf" containerName="maas-api" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308493 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f0be845-0d78-4855-99e8-fceee5d1db1e" containerName="payload-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308501 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="01b77f4c-be6f-4bec-ad2c-90f342187369" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308510 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="e07e4d29-303a-4c97-8715-e9804db8d803" containerName="payload-pre-processing" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308519 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="719d7203-6520-42c5-9c07-0837c6561a8d" containerName="main" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308527 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c4382fe-0417-4b17-be3f-b4bd257b8876" containerName="istio-proxy" Oct 31 00:32:16.312351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.308535 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:16.315499 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.313875 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.339651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.339561 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 00:32:16.340564 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.340538 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 00:32:16.340707 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.340690 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 00:32:16.340878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.340861 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k4pdx\"" Oct 31 00:32:16.341546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.341518 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:16.367881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.367838 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gjttl\" (UniqueName: \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.368345 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.368273 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.368465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.368410 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.473911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.473851 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.473911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.473910 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.474203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.474069 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gjttl\" (UniqueName: \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.477444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.477412 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.483170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.483139 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.488763 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.488729 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gjttl\" (UniqueName: \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\") pod \"maas-controller-7b8bb6f8d-hkrzz\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:16.667599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:16.667243 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:17.550908 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.539294 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:17.662746 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.651435 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:32:17.957362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.957312 6234 generic.go:356] "Generic (PLEG): container finished" podID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerID="a59651bf71f42f4ef4e0b0c90a7266522edc271235de1c07584edda77efeff0e" exitCode=0 Oct 31 00:32:17.957723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.957697 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerDied","Data":"a59651bf71f42f4ef4e0b0c90a7266522edc271235de1c07584edda77efeff0e"} Oct 31 00:32:17.957976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.957957 6234 scope.go:122] "RemoveContainer" containerID="47a153db7ef4906f494be395424f3eba88ac73925e7dde12d137e8c8818240c3" Oct 31 00:32:17.967604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.967576 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" event={"ID":"086cb0e7-5d56-419f-85e9-85a52b57da72","Type":"ContainerStarted","Data":"2d9132e1bbec04471163e4ed431dbea7f19bbf496888481372084dbbced84355"} Oct 31 00:32:17.976381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.976353 6234 generic.go:356] "Generic (PLEG): container finished" podID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerID="c1ecfe013134fa8eab807008b4d3d85f866e7dc2c53609c274be2869f623465d" exitCode=0 Oct 31 00:32:17.976586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:17.976566 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" event={"ID":"c46f2326-e7b7-4758-a19e-eaa02debd40e","Type":"ContainerDied","Data":"c1ecfe013134fa8eab807008b4d3d85f866e7dc2c53609c274be2869f623465d"} Oct 31 00:32:18.433809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.431714 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:32:18.565213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.565179 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.566563 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.566539 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.566849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.566828 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.567042 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.567025 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.567201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.567162 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.567325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.567310 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.567476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.567460 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.567592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.567578 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.569318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.569297 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\") pod \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\" (UID: \"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c\") " Oct 31 00:32:18.570272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.566470 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:18.570398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.568966 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.570495 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.570076 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.570576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.570243 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.570719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.570699 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.655328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.655279 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.656599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.656559 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "kube-api-access-cf42w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:18.657835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.657766 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:18.659110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.659066 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token" pod "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" (UID: "d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672458 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672494 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672510 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf42w\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-kube-api-access-cf42w\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672528 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672544 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672560 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672575 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672590 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.672645 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.672604 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.714587 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.714551 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:32:18.773919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.773873 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.773954 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.773994 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774023 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774063 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774118 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774104 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774137 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774192 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774193 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.774412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774306 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\") pod \"c46f2326-e7b7-4758-a19e-eaa02debd40e\" (UID: \"c46f2326-e7b7-4758-a19e-eaa02debd40e\") " Oct 31 00:32:18.774412 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774317 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774429 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774586 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774748 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774768 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774797 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774814 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.774860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.774849 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:32:18.777415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.777382 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:32:18.777525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.777427 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:32:18.777525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.777489 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "kube-api-access-m9mvt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:18.777611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.777572 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token" pod "c46f2326-e7b7-4758-a19e-eaa02debd40e" (UID: "c46f2326-e7b7-4758-a19e-eaa02debd40e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:18.875712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.875674 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c46f2326-e7b7-4758-a19e-eaa02debd40e-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.875712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.875706 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.875712 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.875718 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.875976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.875729 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9mvt\" (UniqueName: \"kubernetes.io/projected/c46f2326-e7b7-4758-a19e-eaa02debd40e-kube-api-access-m9mvt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.875976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.875738 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c46f2326-e7b7-4758-a19e-eaa02debd40e-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:18.982986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.982956 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" Oct 31 00:32:18.982986 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.982964 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7" event={"ID":"d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c","Type":"ContainerDied","Data":"e92cdfd32e78a4eaed54dbe9b0a62e8f39221129c04da50231abcb74bc3b640a"} Oct 31 00:32:18.983251 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.983041 6234 scope.go:122] "RemoveContainer" containerID="a59651bf71f42f4ef4e0b0c90a7266522edc271235de1c07584edda77efeff0e" Oct 31 00:32:18.985105 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.985082 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" event={"ID":"c46f2326-e7b7-4758-a19e-eaa02debd40e","Type":"ContainerDied","Data":"bf52e01be9d8e3f98526121075e20afd57d4ae9d97aaea4c406ebe094218d7ed"} Oct 31 00:32:18.985221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.985091 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz" Oct 31 00:32:18.995186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:18.995164 6234 scope.go:122] "RemoveContainer" containerID="c1ecfe013134fa8eab807008b4d3d85f866e7dc2c53609c274be2869f623465d" Oct 31 00:32:19.009002 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:19.008975 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:32:19.013557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:19.013534 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-nrkm7"] Oct 31 00:32:19.026468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:19.026445 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:32:19.030143 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:19.030124 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-7b2341-openshift-default-7784b78dcc-mk8cz"] Oct 31 00:32:20.082197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:20.082150 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" path="/var/lib/kubelet/pods/c46f2326-e7b7-4758-a19e-eaa02debd40e/volumes" Oct 31 00:32:20.082739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:20.082714 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" path="/var/lib/kubelet/pods/d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c/volumes" Oct 31 00:32:20.999476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:20.999441 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" event={"ID":"086cb0e7-5d56-419f-85e9-85a52b57da72","Type":"ContainerStarted","Data":"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18"} Oct 31 00:32:20.999692 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:20.999597 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:21.022017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:21.021967 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" podStartSLOduration=2.345700389 podStartE2EDuration="5.021952606s" totalImagesPullingTime="2.676252217s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:21.02108834 +0000 UTC m=+2263.054184410" watchObservedRunningTime="2026-10-31 00:32:21.021952606 +0000 UTC m=+2263.055048673" Oct 31 00:32:32.007892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:32.007856 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:33.160442 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:32:33.160404 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:32:51.560715 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.560663 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:51.561509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.560934 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" podUID="086cb0e7-5d56-419f-85e9-85a52b57da72" containerName="manager" containerID="cri-o://11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18" gracePeriod=30 Oct 31 00:32:51.820523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.820468 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:51.904261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.904228 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\") pod \"086cb0e7-5d56-419f-85e9-85a52b57da72\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " Oct 31 00:32:51.904436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.904337 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\" (UniqueName: \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\") pod \"086cb0e7-5d56-419f-85e9-85a52b57da72\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " Oct 31 00:32:51.904436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.904371 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\") pod \"086cb0e7-5d56-419f-85e9-85a52b57da72\" (UID: \"086cb0e7-5d56-419f-85e9-85a52b57da72\") " Oct 31 00:32:51.906529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.906495 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs" pod "086cb0e7-5d56-419f-85e9-85a52b57da72" (UID: "086cb0e7-5d56-419f-85e9-85a52b57da72"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:51.906654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.906592 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl" pod "086cb0e7-5d56-419f-85e9-85a52b57da72" (UID: "086cb0e7-5d56-419f-85e9-85a52b57da72"). InnerVolumeSpecName "kube-api-access-gjttl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:32:51.906654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:51.906634 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert" pod "086cb0e7-5d56-419f-85e9-85a52b57da72" (UID: "086cb0e7-5d56-419f-85e9-85a52b57da72"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:32:52.005262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.005220 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gjttl\" (UniqueName: \"kubernetes.io/projected/086cb0e7-5d56-419f-85e9-85a52b57da72-kube-api-access-gjttl\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.005262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.005255 6234 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-webhook-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.005262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.005264 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/086cb0e7-5d56-419f-85e9-85a52b57da72-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:32:52.148759 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.148727 6234 generic.go:356] "Generic (PLEG): container finished" podID="086cb0e7-5d56-419f-85e9-85a52b57da72" containerID="11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18" exitCode=0 Oct 31 00:32:52.148946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.148775 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" Oct 31 00:32:52.148946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.148812 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" event={"ID":"086cb0e7-5d56-419f-85e9-85a52b57da72","Type":"ContainerDied","Data":"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18"} Oct 31 00:32:52.148946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.148867 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hkrzz" event={"ID":"086cb0e7-5d56-419f-85e9-85a52b57da72","Type":"ContainerDied","Data":"2d9132e1bbec04471163e4ed431dbea7f19bbf496888481372084dbbced84355"} Oct 31 00:32:52.148946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.148889 6234 scope.go:122] "RemoveContainer" containerID="11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18" Oct 31 00:32:52.157539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.157518 6234 scope.go:122] "RemoveContainer" containerID="11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18" Oct 31 00:32:52.157844 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:32:52.157813 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18\": container with ID starting with 11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18 not found: ID does not exist" containerID="11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18" Oct 31 00:32:52.157953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.157852 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18"} err="failed to get container status \"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18\": rpc error: code = NotFound desc = could not find container \"11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18\": container with ID starting with 11e982ad98a3693cf8b67d98f42e8cc690cf29cc42120dd949cba18f59abba18 not found: ID does not exist" Oct 31 00:32:52.170065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.170041 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:52.174807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:52.174767 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hkrzz"] Oct 31 00:32:53.409147 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409112 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hb7vk"] Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409508 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409519 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9a8230f-61fd-42ed-aa40-b9fc374be85e" containerName="istio-proxy" Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409539 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409544 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409561 6234 cpu_manager.go:414] "containerMap: removing container" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:53.409576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409569 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b447616e-0dc2-4299-b91a-e4b22ee30867" containerName="cleanup" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409586 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409595 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409605 6234 cpu_manager.go:414] "containerMap: removing container" podUID="086cb0e7-5d56-419f-85e9-85a52b57da72" containerName="manager" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409612 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="086cb0e7-5d56-419f-85e9-85a52b57da72" containerName="manager" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409625 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409632 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409695 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9c514e69-2fcc-4c62-a390-eddc0863076d" containerName="cleanup" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409707 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409716 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d9bf53c8-a0b5-44ed-bffe-a2aca9b48b5c" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409723 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="086cb0e7-5d56-419f-85e9-85a52b57da72" containerName="manager" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409730 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c46f2326-e7b7-4758-a19e-eaa02debd40e" containerName="istio-proxy" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409735 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="da0eacb9-63e0-4721-b3a1-d91e386cccbe" containerName="cleanup" Oct 31 00:32:53.409833 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.409741 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6773cff0-24af-4eaf-a2e3-7cab7531ef7a" containerName="cleanup" Oct 31 00:32:53.414223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.414205 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.415841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.415812 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.415963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.415939 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mrwtp\" (UniqueName: \"kubernetes.io/projected/2cccc0c2-9727-4cd8-b458-87d18e8caf72-kube-api-access-mrwtp\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.416034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.415974 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.420974 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.420953 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\"" Oct 31 00:32:53.421418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.420992 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-k4pdx\"" Oct 31 00:32:53.421418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.420992 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\"" Oct 31 00:32:53.422415 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.422395 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\"" Oct 31 00:32:53.424570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.424547 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hb7vk"] Oct 31 00:32:53.517078 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.517044 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.517261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.517112 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mrwtp\" (UniqueName: \"kubernetes.io/projected/2cccc0c2-9727-4cd8-b458-87d18e8caf72-kube-api-access-mrwtp\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.517261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.517143 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.519665 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.519643 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-metrics-certs\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.519804 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.519693 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/2cccc0c2-9727-4cd8-b458-87d18e8caf72-webhook-cert\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.530076 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.530053 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mrwtp\" (UniqueName: \"kubernetes.io/projected/2cccc0c2-9727-4cd8-b458-87d18e8caf72-kube-api-access-mrwtp\") pod \"maas-controller-7b8bb6f8d-hb7vk\" (UID: \"2cccc0c2-9727-4cd8-b458-87d18e8caf72\") " pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:53.728557 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:53.728518 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:54.009725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:54.009700 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-7b8bb6f8d-hb7vk"] Oct 31 00:32:54.011601 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:32:54.011566 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2cccc0c2_9727_4cd8_b458_87d18e8caf72.slice/crio-7b17f73e00e4a6f8338fbe2341fe9389c478b7d4d4967570a1550e4fb7b0a679 WatchSource:0}: Error finding container 7b17f73e00e4a6f8338fbe2341fe9389c478b7d4d4967570a1550e4fb7b0a679: Status 404 returned error can't find the container with id 7b17f73e00e4a6f8338fbe2341fe9389c478b7d4d4967570a1550e4fb7b0a679 Oct 31 00:32:54.080179 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:54.080144 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="086cb0e7-5d56-419f-85e9-85a52b57da72" path="/var/lib/kubelet/pods/086cb0e7-5d56-419f-85e9-85a52b57da72/volumes" Oct 31 00:32:54.160971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:54.160934 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" event={"ID":"2cccc0c2-9727-4cd8-b458-87d18e8caf72","Type":"ContainerStarted","Data":"7b17f73e00e4a6f8338fbe2341fe9389c478b7d4d4967570a1550e4fb7b0a679"} Oct 31 00:32:55.167687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:55.167654 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" event={"ID":"2cccc0c2-9727-4cd8-b458-87d18e8caf72","Type":"ContainerStarted","Data":"15e51c003aad3050ada7826e7e51049129af35557f0ac43b8dc0d0854a303d21"} Oct 31 00:32:55.168135 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:55.167801 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:32:55.195507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:32:55.195455 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" podStartSLOduration=1.765906059 podStartE2EDuration="2.195441017s" totalImagesPullingTime="429.534958ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:32:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:32:55.192894637 +0000 UTC m=+2297.225990708" watchObservedRunningTime="2026-10-31 00:32:55.195441017 +0000 UTC m=+2297.228537087" Oct 31 00:33:03.160840 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:33:03.160802 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:33:06.175278 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:33:06.175246 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-7b8bb6f8d-hb7vk" Oct 31 00:33:33.161144 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:33:33.161100 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:34:03.161500 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:34:03.161413 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:34:33.160460 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:34:33.160421 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:34:51.025477 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:34:51.025441 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb"] Oct 31 00:34:51.030566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:34:51.030539 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-84bedf56-f89rb"] Oct 31 00:34:52.083175 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:34:52.083141 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e59b72ac-f0b8-4a70-b5c9-7ce473faba1c" path="/var/lib/kubelet/pods/e59b72ac-f0b8-4a70-b5c9-7ce473faba1c/volumes" Oct 31 00:35:03.163910 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:35:03.163867 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:35:33.189656 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:35:33.189572 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:35:56.407082 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:35:56.407040 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:35:56.407802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:35:56.407649 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:35:56.452758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:35:56.452316 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:35:56.452758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:35:56.452380 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:36:03.160446 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:36:03.160403 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:36:26.366738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:36:26.366617 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:36:26.366738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:36:26.366704 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:36:33.160507 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:36:33.160468 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:36:42.742289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:36:42.742257 6234 scope.go:122] "RemoveContainer" containerID="ef0081b8db8b57e5b263b268bef796e22bf27f565140ed232b5770901dd396c7" Oct 31 00:36:42.750931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:36:42.750910 6234 scope.go:122] "RemoveContainer" containerID="e6a8596e1129ccfd867c49f459c4b0ed0e58ebc6af1d99470363d14fd222eadd" Oct 31 00:37:03.161333 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:37:03.161247 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:37:33.161055 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:37:33.161018 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:38:03.161124 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:38:03.161079 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:38:33.161220 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:38:33.161111 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:39:03.160526 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:39:03.160481 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:39:33.160951 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:39:33.160912 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:40:03.161194 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:40:03.161112 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:40:33.160561 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:40:33.160520 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:41:03.161278 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:41:03.161238 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:41:33.161016 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:41:33.160926 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:42:03.161102 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:03.161054 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:42:33.160637 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:33.160589 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:42:44.369940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:44.369901 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:44.370586 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:44.370535 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:44.785275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:44.785008 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:42:44.785275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:44.785079 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:42:51.780828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.780731 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:42:51.781581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.780965 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" containerID="cri-o://86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05" gracePeriod=30 Oct 31 00:42:51.787385 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.787354 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:42:51.787678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.787626 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" containerID="cri-o://3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f" gracePeriod=30 Oct 31 00:42:51.789984 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.789952 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:42:51.790197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.790177 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" containerID="cri-o://d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009" gracePeriod=30 Oct 31 00:42:51.791219 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.791162 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:42:51.791447 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:51.791406 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" containerID="cri-o://0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e" gracePeriod=30 Oct 31 00:42:55.361285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:55.361242 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-8x82m container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:42:55.361686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:55.361313 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:55.397989 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:55.397958 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:42:55.398168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:55.398015 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:42:57.056691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.056637 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:42:57.086332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.086308 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:42:57.122460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.122436 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:42:57.126840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.126820 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:42:57.182956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.182928 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.182956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.182973 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183010 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183037 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183081 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183105 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183144 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183179 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183214 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183242 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183260 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183308 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.183351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183339 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183381 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183426 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183453 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183483 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183510 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183522 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183540 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183530 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183572 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183599 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183639 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183665 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183688 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183708 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183724 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\") pod \"20ea1c03-d953-4442-a39e-087b6af667d3\" (UID: \"20ea1c03-d953-4442-a39e-087b6af667d3\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183745 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183773 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183827 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183868 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183878 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183907 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\") pod \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\" (UID: \"5cb52220-22f0-40cf-97c5-a7e9534d4c21\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183936 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.183969 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.184005 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184022 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184056 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184091 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\") pod \"c6c51501-e585-422b-88c3-93839313cc78\" (UID: \"c6c51501-e585-422b-88c3-93839313cc78\") " Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184099 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184123 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\") pod \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\" (UID: \"cf6c8532-d777-42ec-a8ff-11f6e683f0ac\") " Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184521 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184545 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184560 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c6c51501-e585-422b-88c3-93839313cc78-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184575 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.185487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.184590 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.186081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.186041 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.187071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.186342 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.187071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.186547 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.187071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.186555 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.187071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.186767 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.188724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.187801 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.188724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.187856 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.188724 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.187940 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.190094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.189807 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.190094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.189871 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:42:57.190582 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.190523 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.190582 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.190540 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.190873 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.190850 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.191328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.191180 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.191328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.191249 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.191328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.191303 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.199250 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199204 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.199378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.199378 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199353 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "kube-api-access-k7kzp". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.199504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199441 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.199566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199526 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:42:57.199566 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199537 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.199718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199689 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.199817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199733 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5" pod "5cb52220-22f0-40cf-97c5-a7e9534d4c21" (UID: "5cb52220-22f0-40cf-97c5-a7e9534d4c21"). InnerVolumeSpecName "kube-api-access-wssh5". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.199817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199742 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.199946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199923 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "kube-api-access-4jcqz". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.200018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.199938 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token" pod "20ea1c03-d953-4442-a39e-087b6af667d3" (UID: "20ea1c03-d953-4442-a39e-087b6af667d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.200212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.200192 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.200259 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.200237 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w" pod "c6c51501-e585-422b-88c3-93839313cc78" (UID: "c6c51501-e585-422b-88c3-93839313cc78"). InnerVolumeSpecName "kube-api-access-7cf2w". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.200510 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.200487 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:42:57.202917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.202894 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token" pod "cf6c8532-d777-42ec-a8ff-11f6e683f0ac" (UID: "cf6c8532-d777-42ec-a8ff-11f6e683f0ac"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:42:57.285218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285179 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c6c51501-e585-422b-88c3-93839313cc78-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285211 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285220 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285230 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4jcqz\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-kube-api-access-4jcqz\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285239 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285248 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20ea1c03-d953-4442-a39e-087b6af667d3-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285256 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285265 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wssh5\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-kube-api-access-wssh5\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285273 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20ea1c03-d953-4442-a39e-087b6af667d3-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285281 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285289 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285302 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285309 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285317 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285325 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285333 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285342 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20ea1c03-d953-4442-a39e-087b6af667d3-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285353 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285361 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5cb52220-22f0-40cf-97c5-a7e9534d4c21-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285368 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285377 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285385 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5cb52220-22f0-40cf-97c5-a7e9534d4c21-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285393 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285402 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c6c51501-e585-422b-88c3-93839313cc78-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285411 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7kzp\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-kube-api-access-k7kzp\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285418 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285426 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7cf2w\" (UniqueName: \"kubernetes.io/projected/c6c51501-e585-422b-88c3-93839313cc78-kube-api-access-7cf2w\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285434 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285442 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285449 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20ea1c03-d953-4442-a39e-087b6af667d3-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.285461 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.285458 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cf6c8532-d777-42ec-a8ff-11f6e683f0ac-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:42:57.430650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.430564 6234 generic.go:356] "Generic (PLEG): container finished" podID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerID="0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e" exitCode=0 Oct 31 00:42:57.430650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.430624 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" Oct 31 00:42:57.430892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.430650 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" event={"ID":"5cb52220-22f0-40cf-97c5-a7e9534d4c21","Type":"ContainerDied","Data":"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e"} Oct 31 00:42:57.430892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.430725 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m" event={"ID":"5cb52220-22f0-40cf-97c5-a7e9534d4c21","Type":"ContainerDied","Data":"a835fb4819c5dac0f8200a6a6bfaa31d5d0dcbf4fca3e0be6b9eb030705d2f37"} Oct 31 00:42:57.430892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.430760 6234 scope.go:122] "RemoveContainer" containerID="0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e" Oct 31 00:42:57.432041 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.432016 6234 generic.go:356] "Generic (PLEG): container finished" podID="20ea1c03-d953-4442-a39e-087b6af667d3" containerID="d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009" exitCode=0 Oct 31 00:42:57.432126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.432056 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" Oct 31 00:42:57.432126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.432106 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" event={"ID":"20ea1c03-d953-4442-a39e-087b6af667d3","Type":"ContainerDied","Data":"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009"} Oct 31 00:42:57.432205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.432156 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn" event={"ID":"20ea1c03-d953-4442-a39e-087b6af667d3","Type":"ContainerDied","Data":"650461026ced6d8cadab4ff2c49a3bca67e0435ffaad5f9ab375b1e8b54ffcdb"} Oct 31 00:42:57.433679 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.433654 6234 generic.go:356] "Generic (PLEG): container finished" podID="c6c51501-e585-422b-88c3-93839313cc78" containerID="86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05" exitCode=0 Oct 31 00:42:57.433825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.433696 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" Oct 31 00:42:57.433825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.433729 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" event={"ID":"c6c51501-e585-422b-88c3-93839313cc78","Type":"ContainerDied","Data":"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05"} Oct 31 00:42:57.433825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.433772 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw" event={"ID":"c6c51501-e585-422b-88c3-93839313cc78","Type":"ContainerDied","Data":"f95647833d5c2da5abf2bdacc31dd82f9603234f0d5eacbfcaa74f552ffa2f87"} Oct 31 00:42:57.435377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.435286 6234 generic.go:356] "Generic (PLEG): container finished" podID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerID="3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f" exitCode=0 Oct 31 00:42:57.435377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.435324 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" event={"ID":"cf6c8532-d777-42ec-a8ff-11f6e683f0ac","Type":"ContainerDied","Data":"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f"} Oct 31 00:42:57.435377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.435358 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" Oct 31 00:42:57.435687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.435363 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw" event={"ID":"cf6c8532-d777-42ec-a8ff-11f6e683f0ac","Type":"ContainerDied","Data":"183f8a736f170e597ec0d6e4d5bf99c71dbea0c2c1270e3204be9a0e9a0231d8"} Oct 31 00:42:57.440329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440300 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:42:57.440905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440886 6234 cpu_manager.go:414] "containerMap: removing container" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" Oct 31 00:42:57.440980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440907 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" Oct 31 00:42:57.440980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440934 6234 cpu_manager.go:414] "containerMap: removing container" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" Oct 31 00:42:57.440980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440941 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" Oct 31 00:42:57.440980 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440976 6234 cpu_manager.go:414] "containerMap: removing container" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.440984 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441001 6234 cpu_manager.go:414] "containerMap: removing container" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441008 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441113 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="c6c51501-e585-422b-88c3-93839313cc78" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441129 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441141 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" containerName="istio-proxy" Oct 31 00:42:57.441212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.441151 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" containerName="istio-proxy" Oct 31 00:42:57.451829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.446240 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.451829 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.451644 6234 scope.go:122] "RemoveContainer" containerID="0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e" Oct 31 00:42:57.452199 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:57.452170 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e\": container with ID starting with 0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e not found: ID does not exist" containerID="0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e" Oct 31 00:42:57.452267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.452215 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e"} err="failed to get container status \"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e\": rpc error: code = NotFound desc = could not find container \"0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e\": container with ID starting with 0b691db811b175982bbf5e8a8726a87924f9ed9f404e5013d12a774acb0b735e not found: ID does not exist" Oct 31 00:42:57.452267 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.452243 6234 scope.go:122] "RemoveContainer" containerID="d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009" Oct 31 00:42:57.462466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.461348 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:42:57.479379 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.479356 6234 scope.go:122] "RemoveContainer" containerID="d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009" Oct 31 00:42:57.479753 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:57.479724 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009\": container with ID starting with d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009 not found: ID does not exist" containerID="d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009" Oct 31 00:42:57.479897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.479765 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009"} err="failed to get container status \"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009\": rpc error: code = NotFound desc = could not find container \"d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009\": container with ID starting with d7049622523403cd4562bc214c39abc447d995c33fb0efc22e41a98a0a1f4009 not found: ID does not exist" Oct 31 00:42:57.479897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.479817 6234 scope.go:122] "RemoveContainer" containerID="86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05" Oct 31 00:42:57.490000 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.489889 6234 scope.go:122] "RemoveContainer" containerID="86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05" Oct 31 00:42:57.490274 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:57.490253 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05\": container with ID starting with 86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05 not found: ID does not exist" containerID="86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05" Oct 31 00:42:57.490370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.490286 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05"} err="failed to get container status \"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05\": rpc error: code = NotFound desc = could not find container \"86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05\": container with ID starting with 86da977fdb31d5145b91fed066c166630df2e0b8f0bf88dd53a9cb2e1d04fe05 not found: ID does not exist" Oct 31 00:42:57.490370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.490310 6234 scope.go:122] "RemoveContainer" containerID="3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f" Oct 31 00:42:57.500287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.500266 6234 scope.go:122] "RemoveContainer" containerID="3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f" Oct 31 00:42:57.500702 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:42:57.500684 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f\": container with ID starting with 3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f not found: ID does not exist" containerID="3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f" Oct 31 00:42:57.500770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.500717 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f"} err="failed to get container status \"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f\": rpc error: code = NotFound desc = could not find container \"3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f\": container with ID starting with 3fa23f42dec68bc3b9bb08c702385ead02a4f85830ba5b7404c4e9b75918a05f not found: ID does not exist" Oct 31 00:42:57.504468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.504444 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:42:57.517608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.517578 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-9rmfw"] Oct 31 00:42:57.535679 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.535653 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-3a9c50-serving-cert\"" Oct 31 00:42:57.570208 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.570176 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:42:57.576489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.576456 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-wzmnn"] Oct 31 00:42:57.591080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.591048 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.591238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.591115 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.591238 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.591211 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2b79t\" (UniqueName: \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.591347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.591240 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.594308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.594277 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:42:57.601488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.601458 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-5jxfw"] Oct 31 00:42:57.628736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.628703 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:42:57.635406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.635380 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-8x82m"] Oct 31 00:42:57.652164 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.652139 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:42:57.657207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.657188 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.669628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.669606 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-3a9c50\"" Oct 31 00:42:57.671493 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.671471 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-367eca3c-dockercfg-kh2nc\"" Oct 31 00:42:57.674034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.674010 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:42:57.681598 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.681532 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:42:57.685071 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.685051 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.691734 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.691706 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.691848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.691762 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2b79t\" (UniqueName: \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.691848 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.691813 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.691940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.691886 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.692961 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.692940 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.695112 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.695094 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.695197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.695097 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.703354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.703331 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:42:57.708320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.708301 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2b79t\" (UniqueName: \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\") pod \"maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.783711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.783657 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:57.792960 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.792925 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.793060 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.792998 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ltm2s\" (UniqueName: \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.793167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.793141 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.793232 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.793185 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77v2k\" (UniqueName: \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.894382 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.894348 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ltm2s\" (UniqueName: \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.894577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.894414 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.894577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.894445 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77v2k\" (UniqueName: \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.894577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.894485 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.895430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.895404 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.895481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.895404 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.909197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.909172 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ltm2s\" (UniqueName: \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\") pod \"payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.909329 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.909292 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-77v2k\" (UniqueName: \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\") pod \"payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:57.970107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.970066 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:57.998796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:57.998752 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:58.067752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.067680 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:42:58.096824 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.094445 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:42:58.097633 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.097600 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20ea1c03-d953-4442-a39e-087b6af667d3" path="/var/lib/kubelet/pods/20ea1c03-d953-4442-a39e-087b6af667d3/volumes" Oct 31 00:42:58.098465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.098434 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5cb52220-22f0-40cf-97c5-a7e9534d4c21" path="/var/lib/kubelet/pods/5cb52220-22f0-40cf-97c5-a7e9534d4c21/volumes" Oct 31 00:42:58.099181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.099155 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c6c51501-e585-422b-88c3-93839313cc78" path="/var/lib/kubelet/pods/c6c51501-e585-422b-88c3-93839313cc78/volumes" Oct 31 00:42:58.099601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.099583 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf6c8532-d777-42ec-a8ff-11f6e683f0ac" path="/var/lib/kubelet/pods/cf6c8532-d777-42ec-a8ff-11f6e683f0ac/volumes" Oct 31 00:42:58.258503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.258475 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:42:58.259567 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:42:58.259542 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75eade7a_2423_4cf6_a8c0_4325cbaa7da3.slice/crio-94d4c23d0ddcf424f12c8498af178aa30f630ba48fa0a5db69dd6d7ca4e4d19c WatchSource:0}: Error finding container 94d4c23d0ddcf424f12c8498af178aa30f630ba48fa0a5db69dd6d7ca4e4d19c: Status 404 returned error can't find the container with id 94d4c23d0ddcf424f12c8498af178aa30f630ba48fa0a5db69dd6d7ca4e4d19c Oct 31 00:42:58.286723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.286635 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:42:58.289836 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:42:58.289770 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6bff4577_1cc4_4fb0_b3c4_a0a25bb6c88e.slice/crio-7a0f3dc5a8dd363ece33704a571effda86c8c3a295625346ff208cbed4954a20 WatchSource:0}: Error finding container 7a0f3dc5a8dd363ece33704a571effda86c8c3a295625346ff208cbed4954a20: Status 404 returned error can't find the container with id 7a0f3dc5a8dd363ece33704a571effda86c8c3a295625346ff208cbed4954a20 Oct 31 00:42:58.441168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.441126 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" event={"ID":"75eade7a-2423-4cf6-a8c0-4325cbaa7da3","Type":"ContainerStarted","Data":"2d8fea183770c04efd7cb5441e65e55befce07a48aea798d602440383af1bc5a"} Oct 31 00:42:58.441346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.441184 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" event={"ID":"75eade7a-2423-4cf6-a8c0-4325cbaa7da3","Type":"ContainerStarted","Data":"94d4c23d0ddcf424f12c8498af178aa30f630ba48fa0a5db69dd6d7ca4e4d19c"} Oct 31 00:42:58.441346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.441248 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:42:58.445355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.445323 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" event={"ID":"edda08d0-371d-4c81-9808-7baf0f8c2018","Type":"ContainerStarted","Data":"0222deda6d0a7c3e828e88ca96e59a6c34c2d40ffe7f553a8dca4f204f0ac484"} Oct 31 00:42:58.447122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.447092 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" event={"ID":"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e","Type":"ContainerStarted","Data":"9444b3023711744902a9ed9dd3ac31d80e21bb96d845fb14203cb3d398317ac2"} Oct 31 00:42:58.447231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.447145 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" event={"ID":"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e","Type":"ContainerStarted","Data":"7a0f3dc5a8dd363ece33704a571effda86c8c3a295625346ff208cbed4954a20"} Oct 31 00:42:58.447231 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.447180 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:42:58.466513 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.466453 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" podStartSLOduration=1.466437289 podStartE2EDuration="1.466437289s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:58.466171506 +0000 UTC m=+2900.499267578" watchObservedRunningTime="2026-10-31 00:42:58.466437289 +0000 UTC m=+2900.499533360" Oct 31 00:42:58.494770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:58.494724 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" podStartSLOduration=1.494710467 podStartE2EDuration="1.494710467s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:58.492024521 +0000 UTC m=+2900.525120591" watchObservedRunningTime="2026-10-31 00:42:58.494710467 +0000 UTC m=+2900.527806530" Oct 31 00:42:59.454802 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:59.454754 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" event={"ID":"edda08d0-371d-4c81-9808-7baf0f8c2018","Type":"ContainerStarted","Data":"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363"} Oct 31 00:42:59.455282 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:59.455207 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:42:59.493590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:42:59.493540 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" podStartSLOduration=2.047266268 podStartE2EDuration="2.493527062s" totalImagesPullingTime="446.260794ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:42:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:42:59.492912169 +0000 UTC m=+2901.526008240" watchObservedRunningTime="2026-10-31 00:42:59.493527062 +0000 UTC m=+2901.526623193" Oct 31 00:43:03.162559 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:43:03.162513 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:43:05.464242 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:05.464209 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:43:06.899595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.899554 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6"] Oct 31 00:43:06.903328 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.903305 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.919534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.919512 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-p7q4d\"" Oct 31 00:43:06.939221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.939192 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6"] Oct 31 00:43:06.981390 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981362 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981411 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-49j2j\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-kube-api-access-49j2j\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981561 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981433 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981694 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981580 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981694 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981650 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981691 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981718 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981816 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:06.981998 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:06.981867 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083084 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083140 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083171 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083193 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083244 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083311 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083343 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083392 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-49j2j\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-kube-api-access-49j2j\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083630 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083424 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.083844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083762 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.084116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.083977 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.084116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.084019 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.084116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.084107 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.084399 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.084371 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.085808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.085748 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.086036 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.086018 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.101061 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.101032 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.101355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.101338 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-49j2j\" (UniqueName: \"kubernetes.io/projected/5265ed5b-66c3-43a0-b4ff-f8b70892d133-kube-api-access-49j2j\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-glzl6\" (UID: \"5265ed5b-66c3-43a0-b4ff-f8b70892d133\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.217186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.217155 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:07.513502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.513331 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6"] Oct 31 00:43:07.515958 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:43:07.515929 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5265ed5b_66c3_43a0_b4ff_f8b70892d133.slice/crio-febaefd2d93339c958f99d07b73bfdd85e892fff6a94e8f73d20e94e6a6e5ee1 WatchSource:0}: Error finding container febaefd2d93339c958f99d07b73bfdd85e892fff6a94e8f73d20e94e6a6e5ee1: Status 404 returned error can't find the container with id febaefd2d93339c958f99d07b73bfdd85e892fff6a94e8f73d20e94e6a6e5ee1 Oct 31 00:43:07.518018 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.517985 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:43:07.518120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.518064 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:43:07.518120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:07.518111 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:43:08.501308 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:08.501271 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" event={"ID":"5265ed5b-66c3-43a0-b4ff-f8b70892d133","Type":"ContainerStarted","Data":"396394c908ceeb9d3bd056e680ecaeea97f2a897abb2c5e3d333844315ffb8cf"} Oct 31 00:43:08.501718 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:08.501327 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" event={"ID":"5265ed5b-66c3-43a0-b4ff-f8b70892d133","Type":"ContainerStarted","Data":"febaefd2d93339c958f99d07b73bfdd85e892fff6a94e8f73d20e94e6a6e5ee1"} Oct 31 00:43:08.528205 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:08.528159 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podStartSLOduration=2.5281472799999998 podStartE2EDuration="2.52814728s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:43:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:43:08.526028998 +0000 UTC m=+2910.559125091" watchObservedRunningTime="2026-10-31 00:43:08.52814728 +0000 UTC m=+2910.561243411" Oct 31 00:43:09.217469 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:09.217432 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:09.219064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:09.219035 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:09.219200 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:09.219084 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:09.465476 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:09.465443 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:43:09.465684 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:09.465527 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:43:10.218193 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:10.218161 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:10.218623 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:10.218212 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:11.218465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:11.218406 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:11.218465 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:11.218456 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:12.218314 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:12.218269 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:12.218503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:12.218333 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:13.217541 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:13.217508 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:13.217813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:13.217557 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:14.218424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:14.218386 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-glzl6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body= Oct 31 00:43:14.218871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:14.218442 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" podUID="5265ed5b-66c3-43a0-b4ff-f8b70892d133" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" Oct 31 00:43:15.760163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:15.760129 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:15.760696 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:15.760538 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:15.761570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:43:15.761550 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-glzl6" Oct 31 00:43:33.161595 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:43:33.161491 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:44:03.162207 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:44:03.162163 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:44:05.667703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:05.667667 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:44:05.700272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:05.700225 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:44:06.346753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.346715 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:44:06.347015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.346965 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" podUID="edda08d0-371d-4c81-9808-7baf0f8c2018" containerName="maas-api" containerID="cri-o://5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363" gracePeriod=30 Oct 31 00:44:06.608618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.608553 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:44:06.705747 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.705712 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\") pod \"edda08d0-371d-4c81-9808-7baf0f8c2018\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " Oct 31 00:44:06.706298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.705776 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\" (UniqueName: \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\") pod \"edda08d0-371d-4c81-9808-7baf0f8c2018\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " Oct 31 00:44:06.706298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.705831 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\") pod \"edda08d0-371d-4c81-9808-7baf0f8c2018\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " Oct 31 00:44:06.706298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.705887 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\") pod \"edda08d0-371d-4c81-9808-7baf0f8c2018\" (UID: \"edda08d0-371d-4c81-9808-7baf0f8c2018\") " Oct 31 00:44:06.706298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.706263 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca" pod "edda08d0-371d-4c81-9808-7baf0f8c2018" (UID: "edda08d0-371d-4c81-9808-7baf0f8c2018"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:06.708152 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.708127 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls" pod "edda08d0-371d-4c81-9808-7baf0f8c2018" (UID: "edda08d0-371d-4c81-9808-7baf0f8c2018"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:06.708265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.708173 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t" pod "edda08d0-371d-4c81-9808-7baf0f8c2018" (UID: "edda08d0-371d-4c81-9808-7baf0f8c2018"). InnerVolumeSpecName "kube-api-access-2b79t". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:06.708265 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.708217 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs" pod "edda08d0-371d-4c81-9808-7baf0f8c2018" (UID: "edda08d0-371d-4c81-9808-7baf0f8c2018"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:44:06.806362 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806324 6234 generic.go:356] "Generic (PLEG): container finished" podID="edda08d0-371d-4c81-9808-7baf0f8c2018" containerID="5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363" exitCode=0 Oct 31 00:44:06.806546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806387 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" Oct 31 00:44:06.806546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806415 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" event={"ID":"edda08d0-371d-4c81-9808-7baf0f8c2018","Type":"ContainerDied","Data":"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363"} Oct 31 00:44:06.806546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806471 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl" event={"ID":"edda08d0-371d-4c81-9808-7baf0f8c2018","Type":"ContainerDied","Data":"0222deda6d0a7c3e828e88ca96e59a6c34c2d40ffe7f553a8dca4f204f0ac484"} Oct 31 00:44:06.806546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806497 6234 scope.go:122] "RemoveContainer" containerID="5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363" Oct 31 00:44:06.806757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806735 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.806895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806878 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2b79t\" (UniqueName: \"kubernetes.io/projected/edda08d0-371d-4c81-9808-7baf0f8c2018-kube-api-access-2b79t\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.806956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806901 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edda08d0-371d-4c81-9808-7baf0f8c2018-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.806956 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.806916 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edda08d0-371d-4c81-9808-7baf0f8c2018-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:06.815997 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.815977 6234 scope.go:122] "RemoveContainer" containerID="5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363" Oct 31 00:44:06.816257 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:44:06.816239 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363\": container with ID starting with 5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363 not found: ID does not exist" containerID="5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363" Oct 31 00:44:06.816326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.816267 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363"} err="failed to get container status \"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363\": rpc error: code = NotFound desc = could not find container \"5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363\": container with ID starting with 5c9a2b5bcad9b4accb89811274f742f715ed205c098e3c3a7161ffb70c4f1363 not found: ID does not exist" Oct 31 00:44:06.844565 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.844522 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:44:06.846798 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:06.846756 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-3a9c50-68788db56c-wv9tl"] Oct 31 00:44:08.080430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:08.080396 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edda08d0-371d-4c81-9808-7baf0f8c2018" path="/var/lib/kubelet/pods/edda08d0-371d-4c81-9808-7baf0f8c2018/volumes" Oct 31 00:44:10.668867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:10.668810 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" containerID="cri-o://9444b3023711744902a9ed9dd3ac31d80e21bb96d845fb14203cb3d398317ac2" gracePeriod=25 Oct 31 00:44:10.700751 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:10.700712 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" containerID="cri-o://2d8fea183770c04efd7cb5441e65e55befce07a48aea798d602440383af1bc5a" gracePeriod=25 Oct 31 00:44:19.456326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:19.456239 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:44:19.456845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:19.456252 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:44:29.456669 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:29.456625 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:44:29.457075 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:29.456639 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")" Oct 31 00:44:30.918828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.918775 6234 generic.go:356] "Generic (PLEG): container finished" podID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerID="2d8fea183770c04efd7cb5441e65e55befce07a48aea798d602440383af1bc5a" exitCode=0 Oct 31 00:44:30.918828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.918822 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" event={"ID":"75eade7a-2423-4cf6-a8c0-4325cbaa7da3","Type":"ContainerDied","Data":"2d8fea183770c04efd7cb5441e65e55befce07a48aea798d602440383af1bc5a"} Oct 31 00:44:30.920610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.920585 6234 generic.go:356] "Generic (PLEG): container finished" podID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerID="9444b3023711744902a9ed9dd3ac31d80e21bb96d845fb14203cb3d398317ac2" exitCode=0 Oct 31 00:44:30.920727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.920644 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" event={"ID":"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e","Type":"ContainerDied","Data":"9444b3023711744902a9ed9dd3ac31d80e21bb96d845fb14203cb3d398317ac2"} Oct 31 00:44:30.961999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.961971 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:44:30.990634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:30.990602 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:44:31.048134 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048100 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\") pod \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " Oct 31 00:44:31.048323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048172 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\") pod \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " Oct 31 00:44:31.048323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048210 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\" (UniqueName: \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\") pod \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\" (UID: \"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e\") " Oct 31 00:44:31.048323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048259 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\" (UniqueName: \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\") pod \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\" (UID: \"75eade7a-2423-4cf6-a8c0-4325cbaa7da3\") " Oct 31 00:44:31.048568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048544 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume" pod "6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" (UID: "6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:31.048629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.048558 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume" pod "75eade7a-2423-4cf6-a8c0-4325cbaa7da3" (UID: "75eade7a-2423-4cf6-a8c0-4325cbaa7da3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:44:31.050406 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.050381 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k" pod "6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" (UID: "6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e"). InnerVolumeSpecName "kube-api-access-77v2k". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:31.050511 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.050458 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s" pod "75eade7a-2423-4cf6-a8c0-4325cbaa7da3" (UID: "75eade7a-2423-4cf6-a8c0-4325cbaa7da3"). InnerVolumeSpecName "kube-api-access-ltm2s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:44:31.150203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.150160 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:31.150203 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.150203 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:31.150487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.150220 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-77v2k\" (UniqueName: \"kubernetes.io/projected/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e-kube-api-access-77v2k\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:31.150487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.150235 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ltm2s\" (UniqueName: \"kubernetes.io/projected/75eade7a-2423-4cf6-a8c0-4325cbaa7da3-kube-api-access-ltm2s\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:44:31.926201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.926174 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" Oct 31 00:44:31.926691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.926215 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p" event={"ID":"75eade7a-2423-4cf6-a8c0-4325cbaa7da3","Type":"ContainerDied","Data":"94d4c23d0ddcf424f12c8498af178aa30f630ba48fa0a5db69dd6d7ca4e4d19c"} Oct 31 00:44:31.926691 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.926278 6234 scope.go:122] "RemoveContainer" containerID="2d8fea183770c04efd7cb5441e65e55befce07a48aea798d602440383af1bc5a" Oct 31 00:44:31.927672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.927645 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" event={"ID":"6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e","Type":"ContainerDied","Data":"7a0f3dc5a8dd363ece33704a571effda86c8c3a295625346ff208cbed4954a20"} Oct 31 00:44:31.927672 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.927658 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8" Oct 31 00:44:31.935322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.935298 6234 scope.go:122] "RemoveContainer" containerID="9444b3023711744902a9ed9dd3ac31d80e21bb96d845fb14203cb3d398317ac2" Oct 31 00:44:31.954726 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.954695 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:44:31.959585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.959559 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-3a9c50-57b5bcd852mrl8"] Oct 31 00:44:31.975500 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.975473 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:44:31.980713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:31.980688 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-single-main-3a9c50-55f4cs6j7p"] Oct 31 00:44:32.080752 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:32.080719 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" path="/var/lib/kubelet/pods/6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e/volumes" Oct 31 00:44:32.081172 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:32.081153 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" path="/var/lib/kubelet/pods/75eade7a-2423-4cf6-a8c0-4325cbaa7da3/volumes" Oct 31 00:44:33.159952 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:44:33.159917 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:44:52.018745 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.018710 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8"] Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019150 6234 cpu_manager.go:414] "containerMap: removing container" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019164 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019179 6234 cpu_manager.go:414] "containerMap: removing container" podUID="edda08d0-371d-4c81-9808-7baf0f8c2018" containerName="maas-api" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019184 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edda08d0-371d-4c81-9808-7baf0f8c2018" containerName="maas-api" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019200 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019205 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019289 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="75eade7a-2423-4cf6-a8c0-4325cbaa7da3" containerName="payload-pre-processing" Oct 31 00:44:52.019301 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019298 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="edda08d0-371d-4c81-9808-7baf0f8c2018" containerName="maas-api" Oct 31 00:44:52.019554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.019308 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6bff4577-1cc4-4fb0-b3c4-a0a25bb6c88e" containerName="payload-processing" Oct 31 00:44:52.023576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.023554 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.043013 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.042981 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8"] Oct 31 00:44:52.081596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.081567 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9"] Oct 31 00:44:52.085220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.085195 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.104774 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.104737 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9"] Oct 31 00:44:52.147810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.147752 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.147992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.147883 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b6beec-0a69-4aa8-b4ab-1dec71762321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.147992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.147933 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.147992 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.147967 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148093 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148164 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148223 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148208 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148244 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148298 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gbkjr\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-kube-api-access-gbkjr\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148376 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148350 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148435 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148518 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148473 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148551 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148579 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148648 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148637 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148665 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.148743 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148693 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.148844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.148739 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f97h8\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-kube-api-access-f97h8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250075 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250116 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250135 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250260 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f97h8\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-kube-api-access-f97h8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250334 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250393 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250375 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b6beec-0a69-4aa8-b4ab-1dec71762321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250419 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250458 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250495 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250526 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250563 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250595 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.250644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250626 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gbkjr\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-kube-api-access-gbkjr\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250667 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250697 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250726 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250811 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250841 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250845 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.250892 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251175 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/46b6beec-0a69-4aa8-b4ab-1dec71762321-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251187 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251168 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251222 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-data\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251434 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.251688 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251583 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-credential-socket\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251825 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251710 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.251877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.251846 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-workload-certs\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.253126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.253103 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.253235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.253101 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.253235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.253157 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-envoy\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.253678 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.253661 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-podinfo\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.264767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.264739 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.265533 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.265511 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-istio-token\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.266025 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.266007 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f97h8\" (UniqueName: \"kubernetes.io/projected/46b6beec-0a69-4aa8-b4ab-1dec71762321-kube-api-access-f97h8\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-br6t8\" (UID: \"46b6beec-0a69-4aa8-b4ab-1dec71762321\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.271263 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.271208 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gbkjr\" (UniqueName: \"kubernetes.io/projected/758f2634-3775-48c4-a5c6-e1eab5e9e48e-kube-api-access-gbkjr\") pod \"maas-default-gateway-openshift-default-8b64f9ff9-znrn9\" (UID: \"758f2634-3775-48c4-a5c6-e1eab5e9e48e\") " pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.337766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.337714 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:52.402569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.402311 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:52.621841 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.621773 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8"] Oct 31 00:44:52.629597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.629562 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:52.629703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.629626 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:52.629703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.629651 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:52.691107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.691081 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9"] Oct 31 00:44:52.694175 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:44:52.694146 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758f2634_3775_48c4_a5c6_e1eab5e9e48e.slice/crio-cd1b76cf1edfd5c72a488bec37a3049913a668189e78d6b236b2f67265e1eb5f WatchSource:0}: Error finding container cd1b76cf1edfd5c72a488bec37a3049913a668189e78d6b236b2f67265e1eb5f: Status 404 returned error can't find the container with id cd1b76cf1edfd5c72a488bec37a3049913a668189e78d6b236b2f67265e1eb5f Oct 31 00:44:52.696392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.696362 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:52.696474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.696426 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:52.696474 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:52.696453 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:44:53.035093 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.035052 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" event={"ID":"46b6beec-0a69-4aa8-b4ab-1dec71762321","Type":"ContainerStarted","Data":"69d40d972dc7da9616901de96dded325b7aab8b79d7c1daac06dee34b24ad310"} Oct 31 00:44:53.035546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.035118 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" event={"ID":"46b6beec-0a69-4aa8-b4ab-1dec71762321","Type":"ContainerStarted","Data":"26788049c0516ba50902f4355627bd08d5d1d779a2086538687a2cd96d302b6d"} Oct 31 00:44:53.036858 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.036823 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" event={"ID":"758f2634-3775-48c4-a5c6-e1eab5e9e48e","Type":"ContainerStarted","Data":"2f9a90d2f9568e0b2e2113a5fb35753778a7ab0bf552d2b0c699e20718619e04"} Oct 31 00:44:53.036993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.036878 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" event={"ID":"758f2634-3775-48c4-a5c6-e1eab5e9e48e","Type":"ContainerStarted","Data":"cd1b76cf1edfd5c72a488bec37a3049913a668189e78d6b236b2f67265e1eb5f"} Oct 31 00:44:53.062530 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.062481 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podStartSLOduration=2.062429897 podStartE2EDuration="2.062429897s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:53.060079298 +0000 UTC m=+3015.093175368" watchObservedRunningTime="2026-10-31 00:44:53.062429897 +0000 UTC m=+3015.095525966" Oct 31 00:44:53.091044 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.090999 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podStartSLOduration=1.090987173 podStartE2EDuration="1.090987173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:44:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:44:53.089332935 +0000 UTC m=+3015.122429030" watchObservedRunningTime="2026-10-31 00:44:53.090987173 +0000 UTC m=+3015.124083237" Oct 31 00:44:53.338033 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.337943 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:53.339490 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.339466 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:44:53.339611 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.339511 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:44:53.403347 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.403311 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:53.404607 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.404582 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:44:53.404739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:53.404636 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:44:54.342687 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:54.342657 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:54.407658 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:54.407626 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:55.046287 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:55.046253 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:44:55.046487 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:55.046314 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:55.047034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:55.047013 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:44:55.047136 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:44:55.047064 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:45:03.161036 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:45:03.160998 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:45:33.160291 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:45:33.160249 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:46:03.161445 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:46:03.161355 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:46:20.462869 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.462833 6234 generic.go:356] "Generic (PLEG): container finished" podID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerID="69d40d972dc7da9616901de96dded325b7aab8b79d7c1daac06dee34b24ad310" exitCode=0 Oct 31 00:46:20.463373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.462908 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" event={"ID":"46b6beec-0a69-4aa8-b4ab-1dec71762321","Type":"ContainerDied","Data":"69d40d972dc7da9616901de96dded325b7aab8b79d7c1daac06dee34b24ad310"} Oct 31 00:46:20.463448 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.463390 6234 scope.go:122] "RemoveContainer" containerID="69d40d972dc7da9616901de96dded325b7aab8b79d7c1daac06dee34b24ad310" Oct 31 00:46:20.464537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.464517 6234 generic.go:356] "Generic (PLEG): container finished" podID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerID="2f9a90d2f9568e0b2e2113a5fb35753778a7ab0bf552d2b0c699e20718619e04" exitCode=0 Oct 31 00:46:20.464615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.464570 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" event={"ID":"758f2634-3775-48c4-a5c6-e1eab5e9e48e","Type":"ContainerDied","Data":"2f9a90d2f9568e0b2e2113a5fb35753778a7ab0bf552d2b0c699e20718619e04"} Oct 31 00:46:20.464716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.464685 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:20.464813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.464771 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:20.464864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.464838 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:20.465064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.465044 6234 scope.go:122] "RemoveContainer" containerID="2f9a90d2f9568e0b2e2113a5fb35753778a7ab0bf552d2b0c699e20718619e04" Oct 31 00:46:20.466504 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.466476 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:20.466569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.466552 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:20.466610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:20.466597 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:21.493398 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:21.493353 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" event={"ID":"758f2634-3775-48c4-a5c6-e1eab5e9e48e","Type":"ContainerStarted","Data":"b0003c59c8efd000b4508a688d7d1b9018cc2098a901154307b9c8207a039fab"} Oct 31 00:46:21.496081 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:21.496051 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" event={"ID":"46b6beec-0a69-4aa8-b4ab-1dec71762321","Type":"ContainerStarted","Data":"9ae332738dc4afb2895892e51dc6c32ff4e7250a1d5a8eb0e15df587073385b1"} Oct 31 00:46:22.342813 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.338871 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:46:22.359335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.358879 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.359335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.358947 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:22.403539 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.403500 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:46:22.404367 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.404344 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:22.404439 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:22.404392 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:23.338649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:23.338609 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:23.339195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:23.338663 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:23.403408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:23.403365 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:23.403599 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:23.403421 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:24.338069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:24.338028 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:24.338253 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:24.338090 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:24.403334 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:24.403284 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:24.403737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:24.403346 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:25.050810 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.049956 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:46:25.051046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.050867 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:46:25.338968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.338870 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:25.339274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.339234 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:25.403430 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.403398 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:25.403953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:25.403919 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:26.338551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:26.338512 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:26.338765 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:26.338584 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:26.403155 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:26.403115 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:26.403370 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:26.403178 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:27.338612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:27.338565 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" start-of-body= Oct 31 00:46:27.339220 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:27.338624 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": dial tcp 10.134.0.165:15021: connect: connection refused" Oct 31 00:46:27.403883 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:27.403834 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" start-of-body= Oct 31 00:46:27.404092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:27.403902 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": dial tcp 10.134.0.166:15021: connect: connection refused" Oct 31 00:46:29.339595 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:29.339557 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:29.340163 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:29.339610 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:29.404629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:29.404588 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:29.404899 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:29.404655 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.385809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:30.384912 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:30.385809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:30.384985 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:30.454965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:30.454916 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:30.455312 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:30.455262 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:31.411275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:31.411225 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:31.411808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:31.411287 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:31.455864 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:31.455824 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:31.456149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:31.456124 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:32.453007 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:32.452965 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:32.453644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:32.453610 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:32.499320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:32.498933 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:32.499320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:32.499003 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.175896 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:46:33.175796 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:46:33.465681 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.465635 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-br6t8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:33.466248 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.465718 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" podUID="46b6beec-0a69-4aa8-b4ab-1dec71762321" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.165:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.540968 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.540924 6234 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-8b64f9ff9-znrn9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body= Oct 31 00:46:33.541180 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.540996 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" podUID="758f2634-3775-48c4-a5c6-e1eab5e9e48e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.166:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" Oct 31 00:46:33.754138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.754046 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:46:33.755335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.755311 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-br6t8" Oct 31 00:46:33.988967 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:33.988936 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:46:34.019903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:34.019807 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-8b64f9ff9-znrn9" Oct 31 00:46:48.748472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.748437 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:46:48.751919 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.751901 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.758442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.758415 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-b0db87-openshift-default-dockercfg-mcng8\"" Oct 31 00:46:48.764277 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.764220 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:46:48.791753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.791715 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.791976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.791768 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.791976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.791816 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.791976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.791929 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-khhkq\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.792139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.791976 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.792139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.792057 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.792139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.792103 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.792139 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.792132 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.792355 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.792261 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.899127 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.899071 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.899914 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.899869 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.899754 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.899921 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900008 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900074 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900065 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-khhkq\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900124 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900181 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900206 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.900321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.900250 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.901116 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.901091 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.905126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.904934 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.905126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.904965 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.905126 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.905122 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.905554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.905530 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.905709 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.905690 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.914749 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.914724 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:48.915313 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:48.915287 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-khhkq\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\") pod \"e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:49.068921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.068828 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:49.354850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.354824 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:46:49.356738 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:46:49.356711 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c998055_2284_4dee_9f86_8a77dade0ccc.slice/crio-cde95d1e51f37d599a645f32d7b5cada4fbcf80ce967fda8768124de53038c2f WatchSource:0}: Error finding container cde95d1e51f37d599a645f32d7b5cada4fbcf80ce967fda8768124de53038c2f: Status 404 returned error can't find the container with id cde95d1e51f37d599a645f32d7b5cada4fbcf80ce967fda8768124de53038c2f Oct 31 00:46:49.359213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.359168 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:49.359304 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.359261 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:49.359350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.359306 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:46:49.668387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.668294 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerStarted","Data":"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a"} Oct 31 00:46:49.668387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.668362 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerStarted","Data":"cde95d1e51f37d599a645f32d7b5cada4fbcf80ce967fda8768124de53038c2f"} Oct 31 00:46:49.694730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:49.694683 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podStartSLOduration=1.694668814 podStartE2EDuration="1.694668814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:49.692745281 +0000 UTC m=+3131.725841353" watchObservedRunningTime="2026-10-31 00:46:49.694668814 +0000 UTC m=+3131.727764936" Oct 31 00:46:50.069368 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:50.069326 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:50.079714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:50.079688 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:50.672916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:50.672881 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:50.673726 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:50.673699 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:46:53.297388 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.297347 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:46:53.301418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.301392 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.336879 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.336844 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-b0db87-serving-cert\"" Oct 31 00:46:53.338610 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.338579 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:46:53.358442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.358405 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mc677\" (UniqueName: \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.358442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.358444 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.358675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.358497 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.358675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.358529 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.459874 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.459835 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.460084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.459896 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.460084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.459946 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mc677\" (UniqueName: \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.460084 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.459966 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.460711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.460624 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.462496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.462473 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.462496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.462485 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.485622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.485594 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mc677\" (UniqueName: \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\") pod \"maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.616637 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.616553 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:53.898072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:53.898025 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:46:53.900025 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:46:53.899995 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod04bf5e20_9251_4962_9ac8_5fb8751fd21a.slice/crio-02d685ef6d7627919b9a8cdb7631f570ec69712042e0da2650cfac2d942051e0 WatchSource:0}: Error finding container 02d685ef6d7627919b9a8cdb7631f570ec69712042e0da2650cfac2d942051e0: Status 404 returned error can't find the container with id 02d685ef6d7627919b9a8cdb7631f570ec69712042e0da2650cfac2d942051e0 Oct 31 00:46:54.694912 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:54.694872 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" event={"ID":"04bf5e20-9251-4962-9ac8-5fb8751fd21a","Type":"ContainerStarted","Data":"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2"} Oct 31 00:46:54.695361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:54.694936 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" event={"ID":"04bf5e20-9251-4962-9ac8-5fb8751fd21a","Type":"ContainerStarted","Data":"02d685ef6d7627919b9a8cdb7631f570ec69712042e0da2650cfac2d942051e0"} Oct 31 00:46:54.695361 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:54.694967 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:46:54.717285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:46:54.717222 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" podStartSLOduration=1.140762009 podStartE2EDuration="1.717202833s" totalImagesPullingTime="576.440824ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:46:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:46:54.715991186 +0000 UTC m=+3136.749087298" watchObservedRunningTime="2026-10-31 00:46:54.717202833 +0000 UTC m=+3136.750298905" Oct 31 00:47:00.705342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:00.705304 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:47:03.165584 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:47:03.165544 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:47:05.674568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.674489 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:05.674568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.674552 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:05.757381 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.757300 6234 generic.go:356] "Generic (PLEG): container finished" podID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerID="6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a" exitCode=0 Oct 31 00:47:05.757534 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.757375 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerDied","Data":"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a"} Oct 31 00:47:05.757731 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.757715 6234 scope.go:122] "RemoveContainer" containerID="6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a" Oct 31 00:47:05.758898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.758865 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:47:05.759016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.758951 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:47:05.759016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:05.759001 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:47:06.770346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:06.770309 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerStarted","Data":"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d"} Oct 31 00:47:07.069551 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:07.069462 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:47:07.069713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:07.069677 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:07.069764 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:07.069724 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:08.069507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:08.069468 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:08.070056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:08.069549 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:09.069525 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:09.069481 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:09.070077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:09.069539 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:10.070053 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:10.070011 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:10.070550 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:10.070072 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:11.070040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:11.070005 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" start-of-body= Oct 31 00:47:11.070228 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:11.070057 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.167:15021/healthz/ready\": dial tcp 10.134.0.167:15021: connect: connection refused" Oct 31 00:47:12.073258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:12.073227 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:47:12.073654 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:12.073580 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:47:12.079466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:47:12.079439 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:47:33.160793 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:47:33.160695 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:48:01.404976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.404939 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:48:01.405464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.405183 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" podUID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" containerName="maas-api" containerID="cri-o://1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2" gracePeriod=30 Oct 31 00:48:01.698801 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.698754 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:48:01.775846 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.775815 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\") pod \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " Oct 31 00:48:01.776064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.775896 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\") pod \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " Oct 31 00:48:01.776064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.775942 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\" (UniqueName: \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\") pod \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " Oct 31 00:48:01.776064 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.776013 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\") pod \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\" (UID: \"04bf5e20-9251-4962-9ac8-5fb8751fd21a\") " Oct 31 00:48:01.776296 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.776271 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca" pod "04bf5e20-9251-4962-9ac8-5fb8751fd21a" (UID: "04bf5e20-9251-4962-9ac8-5fb8751fd21a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:01.778291 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.778260 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677" pod "04bf5e20-9251-4962-9ac8-5fb8751fd21a" (UID: "04bf5e20-9251-4962-9ac8-5fb8751fd21a"). InnerVolumeSpecName "kube-api-access-mc677". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:01.778416 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.778319 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls" pod "04bf5e20-9251-4962-9ac8-5fb8751fd21a" (UID: "04bf5e20-9251-4962-9ac8-5fb8751fd21a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:01.778416 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.778329 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs" pod "04bf5e20-9251-4962-9ac8-5fb8751fd21a" (UID: "04bf5e20-9251-4962-9ac8-5fb8751fd21a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:48:01.877201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.877165 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:01.877201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.877198 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/04bf5e20-9251-4962-9ac8-5fb8751fd21a-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:01.877201 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.877208 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/04bf5e20-9251-4962-9ac8-5fb8751fd21a-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:01.877472 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:01.877217 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mc677\" (UniqueName: \"kubernetes.io/projected/04bf5e20-9251-4962-9ac8-5fb8751fd21a-kube-api-access-mc677\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:02.056284 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.056248 6234 generic.go:356] "Generic (PLEG): container finished" podID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" containerID="1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2" exitCode=0 Oct 31 00:48:02.056467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.056295 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" Oct 31 00:48:02.056467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.056306 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" event={"ID":"04bf5e20-9251-4962-9ac8-5fb8751fd21a","Type":"ContainerDied","Data":"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2"} Oct 31 00:48:02.056467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.056377 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6" event={"ID":"04bf5e20-9251-4962-9ac8-5fb8751fd21a","Type":"ContainerDied","Data":"02d685ef6d7627919b9a8cdb7631f570ec69712042e0da2650cfac2d942051e0"} Oct 31 00:48:02.056467 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.056405 6234 scope.go:122] "RemoveContainer" containerID="1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2" Oct 31 00:48:02.065828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.065805 6234 scope.go:122] "RemoveContainer" containerID="1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2" Oct 31 00:48:02.066101 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:48:02.066081 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2\": container with ID starting with 1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2 not found: ID does not exist" containerID="1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2" Oct 31 00:48:02.066148 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.066111 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2"} err="failed to get container status \"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2\": rpc error: code = NotFound desc = could not find container \"1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2\": container with ID starting with 1c21142be96e0e27424298bad66baff4a807a81f5bfe9a1b96eca947e83d46d2 not found: ID does not exist" Oct 31 00:48:02.121978 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.121946 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:48:02.127129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:02.127097 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-b0db87-88967bcd7-tvjf6"] Oct 31 00:48:03.160744 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:48:03.160708 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:48:04.080549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:04.080517 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" path="/var/lib/kubelet/pods/04bf5e20-9251-4962-9ac8-5fb8751fd21a/volumes" Oct 31 00:48:08.679137 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:08.679103 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:48:08.679655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:08.679322 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" containerID="cri-o://ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d" gracePeriod=30 Oct 31 00:48:12.076158 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:12.076122 6234 patch_prober.go:29] interesting pod/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body= Oct 31 00:48:12.076594 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:12.076202 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:48:13.965213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:13.965187 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:48:14.103917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.103888 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.103940 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.103995 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104176 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:14.104218 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104183 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104218 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:14.104342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104254 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104342 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104291 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104344 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104403 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.104484 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.104461 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\") pod \"6c998055-2284-4dee-9f86-8a77dade0ccc\" (UID: \"6c998055-2284-4dee-9f86-8a77dade0ccc\") " Oct 31 00:48:14.105121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.105098 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.105217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.105126 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.105325 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.105301 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:48:14.105392 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.105374 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:14.105452 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.105409 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:14.106450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.106421 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:48:14.106576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.106551 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "kube-api-access-khhkq". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:14.106823 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.106801 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:48:14.106894 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.106842 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token" pod "6c998055-2284-4dee-9f86-8a77dade0ccc" (UID: "6c998055-2284-4dee-9f86-8a77dade0ccc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:48:14.123069 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.123043 6234 generic.go:356] "Generic (PLEG): container finished" podID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerID="ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d" exitCode=0 Oct 31 00:48:14.123194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.123118 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" Oct 31 00:48:14.123194 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.123122 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerDied","Data":"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d"} Oct 31 00:48:14.123319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.123211 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd" event={"ID":"6c998055-2284-4dee-9f86-8a77dade0ccc","Type":"ContainerDied","Data":"cde95d1e51f37d599a645f32d7b5cada4fbcf80ce967fda8768124de53038c2f"} Oct 31 00:48:14.123319 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.123243 6234 scope.go:122] "RemoveContainer" containerID="ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d" Oct 31 00:48:14.132570 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.132550 6234 scope.go:122] "RemoveContainer" containerID="6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a" Oct 31 00:48:14.140523 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.140506 6234 scope.go:122] "RemoveContainer" containerID="ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d" Oct 31 00:48:14.140821 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:48:14.140802 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d\": container with ID starting with ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d not found: ID does not exist" containerID="ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d" Oct 31 00:48:14.140886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.140832 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d"} err="failed to get container status \"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d\": rpc error: code = NotFound desc = could not find container \"ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d\": container with ID starting with ac5bff8bf4cddae6edec56c0480e85b98589e966e62fc8e47f0e04d0340b730d not found: ID does not exist" Oct 31 00:48:14.140886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.140848 6234 scope.go:122] "RemoveContainer" containerID="6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a" Oct 31 00:48:14.141072 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:48:14.141053 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a\": container with ID starting with 6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a not found: ID does not exist" containerID="6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a" Oct 31 00:48:14.141115 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.141078 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a"} err="failed to get container status \"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a\": rpc error: code = NotFound desc = could not find container \"6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a\": container with ID starting with 6501a11615063e82ba5f84bb5dc4364d939d9b603332896113333af3d925fa5a not found: ID does not exist" Oct 31 00:48:14.153954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.153932 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:48:14.159604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.159580 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-b0db87-openshift-default-5477668b69568cd"] Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205890 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205922 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-khhkq\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-kube-api-access-khhkq\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205940 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205952 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205964 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6c998055-2284-4dee-9f86-8a77dade0ccc-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.205979 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205976 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6c998055-2284-4dee-9f86-8a77dade0ccc-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:14.206283 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:14.205989 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6c998055-2284-4dee-9f86-8a77dade0ccc-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:48:16.081922 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:16.081884 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" path="/var/lib/kubelet/pods/6c998055-2284-4dee-9f86-8a77dade0ccc/volumes" Oct 31 00:48:30.365001 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.364965 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:48:30.365490 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365475 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:48:30.365531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365491 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:48:30.365531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365506 6234 cpu_manager.go:414] "containerMap: removing container" podUID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" containerName="maas-api" Oct 31 00:48:30.365531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365511 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" containerName="maas-api" Oct 31 00:48:30.365703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365610 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:48:30.365703 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.365620 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="04bf5e20-9251-4962-9ac8-5fb8751fd21a" containerName="maas-api" Oct 31 00:48:30.370141 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.370121 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.379964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.379942 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-173a3659-kserve-self-signed-certs\"" Oct 31 00:48:30.381871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.381846 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:48:30.500324 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.500284 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.500820 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.500770 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.501038 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.501020 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cxtgl\" (UniqueName: \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.501225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.501210 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.501661 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.501642 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.501895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.501877 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603321 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603284 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603366 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603390 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603440 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cxtgl\" (UniqueName: \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603487 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603578 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603821 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603814 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603851 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.603987 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.603944 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.605963 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.605938 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.606213 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.606196 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.616744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.616689 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cxtgl\" (UniqueName: \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\") pod \"e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.683870 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.683835 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:30.958653 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:30.958626 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:48:30.961151 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:48:30.961124 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddae40077_77d9_4c17_aed7_3e1139ef7646.slice/crio-82388091fb1900770769396b506111caa8ff99aacc3396c0c4a4cf539386d104 WatchSource:0}: Error finding container 82388091fb1900770769396b506111caa8ff99aacc3396c0c4a4cf539386d104: Status 404 returned error can't find the container with id 82388091fb1900770769396b506111caa8ff99aacc3396c0c4a4cf539386d104 Oct 31 00:48:31.225438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:31.225340 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" event={"ID":"dae40077-77d9-4c17-aed7-3e1139ef7646","Type":"ContainerStarted","Data":"b1e10797c94f3a194cd1328cad814810decb4664440f78fdfbc994fc4c7840dd"} Oct 31 00:48:31.225438 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:31.225388 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" event={"ID":"dae40077-77d9-4c17-aed7-3e1139ef7646","Type":"ContainerStarted","Data":"82388091fb1900770769396b506111caa8ff99aacc3396c0c4a4cf539386d104"} Oct 31 00:48:31.252506 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:31.252452 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" podStartSLOduration=1.252435119 podStartE2EDuration="1.252435119s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:48:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:48:31.251310244 +0000 UTC m=+3233.284406329" watchObservedRunningTime="2026-10-31 00:48:31.252435119 +0000 UTC m=+3233.285531219" Oct 31 00:48:31.684034 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:31.683947 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:33.163808 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:48:33.163277 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:48:40.684904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:40.684868 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:40.697273 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:40.697248 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:41.306062 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:41.306036 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:48:59.307938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:48:59.307843 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:49:03.161346 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:49:03.161304 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:49:14.332881 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.332816 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" podUID="dae40077-77d9-4c17-aed7-3e1139ef7646" containerName="main" containerID="cri-o://b1e10797c94f3a194cd1328cad814810decb4664440f78fdfbc994fc4c7840dd" gracePeriod=45 Oct 31 00:49:14.499292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.499254 6234 generic.go:356] "Generic (PLEG): container finished" podID="dae40077-77d9-4c17-aed7-3e1139ef7646" containerID="b1e10797c94f3a194cd1328cad814810decb4664440f78fdfbc994fc4c7840dd" exitCode=0 Oct 31 00:49:14.499458 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.499329 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" event={"ID":"dae40077-77d9-4c17-aed7-3e1139ef7646","Type":"ContainerDied","Data":"b1e10797c94f3a194cd1328cad814810decb4664440f78fdfbc994fc4c7840dd"} Oct 31 00:49:14.585632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.585565 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:49:14.717161 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717124 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\" (UniqueName: \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717213 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717235 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717272 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717350 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717335 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\" (UniqueName: \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717363 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\") pod \"dae40077-77d9-4c17-aed7-3e1139ef7646\" (UID: \"dae40077-77d9-4c17-aed7-3e1139ef7646\") " Oct 31 00:49:14.717603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717518 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:14.717603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717523 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:14.717755 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717676 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:14.717880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717863 6234 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-home\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.717921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717882 6234 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-tmp-dir\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.717921 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.717892 6234 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-model-cache\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.719619 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.719591 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:49:14.719735 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.719630 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:14.719843 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.719824 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl" pod "dae40077-77d9-4c17-aed7-3e1139ef7646" (UID: "dae40077-77d9-4c17-aed7-3e1139ef7646"). InnerVolumeSpecName "kube-api-access-cxtgl". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:14.819305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.819262 6234 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/dae40077-77d9-4c17-aed7-3e1139ef7646-dshm\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.819305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.819303 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cxtgl\" (UniqueName: \"kubernetes.io/projected/dae40077-77d9-4c17-aed7-3e1139ef7646-kube-api-access-cxtgl\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:14.819521 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:14.819317 6234 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/dae40077-77d9-4c17-aed7-3e1139ef7646-tls-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:15.505930 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:15.505891 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" event={"ID":"dae40077-77d9-4c17-aed7-3e1139ef7646","Type":"ContainerDied","Data":"82388091fb1900770769396b506111caa8ff99aacc3396c0c4a4cf539386d104"} Oct 31 00:49:15.506425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:15.505936 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd" Oct 31 00:49:15.506425 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:15.505957 6234 scope.go:122] "RemoveContainer" containerID="b1e10797c94f3a194cd1328cad814810decb4664440f78fdfbc994fc4c7840dd" Oct 31 00:49:15.533554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:15.533491 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:49:15.536971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:15.536948 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-173a3659-kserve-5c4c5cc445-wfsmd"] Oct 31 00:49:16.086360 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:16.086324 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dae40077-77d9-4c17-aed7-3e1139ef7646" path="/var/lib/kubelet/pods/dae40077-77d9-4c17-aed7-3e1139ef7646/volumes" Oct 31 00:49:33.161120 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:49:33.161087 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:49:34.124706 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.124675 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2"] Oct 31 00:49:34.125108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125094 6234 cpu_manager.go:414] "containerMap: removing container" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:49:34.125160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125109 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:49:34.125160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125141 6234 cpu_manager.go:414] "containerMap: removing container" podUID="dae40077-77d9-4c17-aed7-3e1139ef7646" containerName="main" Oct 31 00:49:34.125160 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125147 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dae40077-77d9-4c17-aed7-3e1139ef7646" containerName="main" Oct 31 00:49:34.125255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125216 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="6c998055-2284-4dee-9f86-8a77dade0ccc" containerName="istio-proxy" Oct 31 00:49:34.125255 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.125229 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="dae40077-77d9-4c17-aed7-3e1139ef7646" containerName="main" Oct 31 00:49:34.129246 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.129229 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.140683 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.140611 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2"] Oct 31 00:49:34.149996 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.149971 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-metrics-certs\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.150190 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.150168 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pgmrc\" (UniqueName: \"kubernetes.io/projected/263c44c2-b7db-4633-b0ee-653c5a35d1af-kube-api-access-pgmrc\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.150323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.150301 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263c44c2-b7db-4633-b0ee-653c5a35d1af-openshift-service-ca\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.150383 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.150347 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-maas-api-tls\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.251207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.251172 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263c44c2-b7db-4633-b0ee-653c5a35d1af-openshift-service-ca\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.251721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.251217 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-maas-api-tls\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.251721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.251278 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-metrics-certs\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.251721 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.251299 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pgmrc\" (UniqueName: \"kubernetes.io/projected/263c44c2-b7db-4633-b0ee-653c5a35d1af-kube-api-access-pgmrc\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.251991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.251963 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/263c44c2-b7db-4633-b0ee-653c5a35d1af-openshift-service-ca\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.254196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.254173 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-maas-api-tls\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.254196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.254192 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/263c44c2-b7db-4633-b0ee-653c5a35d1af-metrics-certs\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.264409 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.264382 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pgmrc\" (UniqueName: \"kubernetes.io/projected/263c44c2-b7db-4633-b0ee-653c5a35d1af-kube-api-access-pgmrc\") pod \"maas-api-8598cb9b7-rptl2\" (UID: \"263c44c2-b7db-4633-b0ee-653c5a35d1af\") " pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.448077 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.448044 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:34.728976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.728950 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2"] Oct 31 00:49:34.731941 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:49:34.731913 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod263c44c2_b7db_4633_b0ee_653c5a35d1af.slice/crio-c6affd85d63c079eec1866dcbbe123a3edbb7b283df7203c42615380264194b2 WatchSource:0}: Error finding container c6affd85d63c079eec1866dcbbe123a3edbb7b283df7203c42615380264194b2: Status 404 returned error can't find the container with id c6affd85d63c079eec1866dcbbe123a3edbb7b283df7203c42615380264194b2 Oct 31 00:49:34.733877 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:34.733854 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:49:35.605008 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:35.604969 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" event={"ID":"263c44c2-b7db-4633-b0ee-653c5a35d1af","Type":"ContainerStarted","Data":"a31ff69ad87f56af8880cccd8ac6ce4990c97bf32948eddcfa9e6f640c108b94"} Oct 31 00:49:35.605466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:35.605024 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" event={"ID":"263c44c2-b7db-4633-b0ee-653c5a35d1af","Type":"ContainerStarted","Data":"c6affd85d63c079eec1866dcbbe123a3edbb7b283df7203c42615380264194b2"} Oct 31 00:49:35.605466 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:35.605091 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:35.626456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:35.626397 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" podStartSLOduration=1.211920418 podStartE2EDuration="1.626381903s" totalImagesPullingTime="414.461485ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:35.625757254 +0000 UTC m=+3297.658853318" watchObservedRunningTime="2026-10-31 00:49:35.626381903 +0000 UTC m=+3297.659477971" Oct 31 00:49:41.614596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.614561 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-8598cb9b7-rptl2" Oct 31 00:49:41.624860 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.624831 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:49:41.628634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.628613 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.633905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.633882 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-fce3e276-openshift-default-dockercfg-vw2zh\"" Oct 31 00:49:41.643065 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.643038 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:49:41.717553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717510 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717645 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717707 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717766 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717836 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c698s\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717866 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.717918 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.717891 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.718121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.718049 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.718274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.718253 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819753 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819709 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819794 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819838 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819885 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819924 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c698s\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.819957 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819952 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.820269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.819976 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.820269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.820027 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.820269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.820083 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.821867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.820977 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.821867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.821244 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.821867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.821512 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.821867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.821806 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.821867 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.821824 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.823046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.823016 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.825604 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.825553 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.835228 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.835195 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c698s\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.837651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.837629 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\") pod \"e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:41.942907 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:41.942872 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:42.225581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.225554 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:49:42.227121 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:49:42.227095 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9826ba3c_5b50_47c2_b3b3_85217f011406.slice/crio-e4f6fedf43d4536c4b09625cfbe93dcbb165635bb1b15d2a01d94d2c76ee702c WatchSource:0}: Error finding container e4f6fedf43d4536c4b09625cfbe93dcbb165635bb1b15d2a01d94d2c76ee702c: Status 404 returned error can't find the container with id e4f6fedf43d4536c4b09625cfbe93dcbb165635bb1b15d2a01d94d2c76ee702c Oct 31 00:49:42.229215 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.229182 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:42.229298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.229246 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:42.229298 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.229273 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:42.638612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.638526 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerStarted","Data":"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568"} Oct 31 00:49:42.638612 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.638576 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerStarted","Data":"e4f6fedf43d4536c4b09625cfbe93dcbb165635bb1b15d2a01d94d2c76ee702c"} Oct 31 00:49:42.663424 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.663372 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podStartSLOduration=1.663359195 podStartE2EDuration="1.663359195s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:42.663277337 +0000 UTC m=+3304.696373407" watchObservedRunningTime="2026-10-31 00:49:42.663359195 +0000 UTC m=+3304.696455264" Oct 31 00:49:42.943751 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:42.943716 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:43.949210 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:43.949176 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:44.654589 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:44.654557 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:44.655496 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:44.655473 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:45.576751 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.576718 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:49:45.581732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.581705 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.590770 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.590747 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-fce3e276\"" Oct 31 00:49:45.592951 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.592928 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:49:45.607197 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.607155 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:49:45.611217 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.611190 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.618936 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.617579 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-fce3e276-dockercfg-dqfsr\"" Oct 31 00:49:45.621104 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.621077 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:49:45.660705 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.660677 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hdfn4\" (UniqueName: \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.660903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.660715 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-564w2\" (UniqueName: \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.660903 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.660890 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.661015 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.660939 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.714690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.712982 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:49:45.725878 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.723904 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.730040 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.729773 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:49:45.731695 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.730637 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-fce3e276-serving-cert\"" Oct 31 00:49:45.761569 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761530 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hdfn4\" (UniqueName: \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.761758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761612 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-564w2\" (UniqueName: \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.761758 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761683 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.761994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761759 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.761994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761829 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.761994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761891 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.761994 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.761924 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.762204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.762026 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z7qx8\" (UniqueName: \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.763269 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.763239 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.763394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.763239 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.777149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.777107 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hdfn4\" (UniqueName: \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\") pod \"payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.784886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.784854 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-564w2\" (UniqueName: \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\") pod \"payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:45.863532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.863445 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.863532 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.863500 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.863766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.863560 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z7qx8\" (UniqueName: \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.863766 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.863638 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.864317 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.864255 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.866226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.866204 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.866326 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.866305 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.876418 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.876395 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z7qx8\" (UniqueName: \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\") pod \"maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:45.897309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.897282 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:45.935708 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:45.928309 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:46.065773 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.065736 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:46.189560 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.189532 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:49:46.191316 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:49:46.191292 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc9a75c3_95ad_433d_8b6b_2e2817c52497.slice/crio-35d06f71b3885da20d49a9f6b2f4083773ddf078e3e8e901a03ff7bd3ef0306c WatchSource:0}: Error finding container 35d06f71b3885da20d49a9f6b2f4083773ddf078e3e8e901a03ff7bd3ef0306c: Status 404 returned error can't find the container with id 35d06f71b3885da20d49a9f6b2f4083773ddf078e3e8e901a03ff7bd3ef0306c Oct 31 00:49:46.222128 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.221135 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:49:46.225419 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:49:46.225392 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda872f8df_b5a6_4038_b55f_866b49962273.slice/crio-97e35f65b0bbfdd0498c93f1e6ebc3a6dfae958f06dd84a07f09a3696a81d295 WatchSource:0}: Error finding container 97e35f65b0bbfdd0498c93f1e6ebc3a6dfae958f06dd84a07f09a3696a81d295: Status 404 returned error can't find the container with id 97e35f65b0bbfdd0498c93f1e6ebc3a6dfae958f06dd84a07f09a3696a81d295 Oct 31 00:49:46.346088 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.346061 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:49:46.349578 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:49:46.349536 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podabef928e_c3b4_491e_9bb0_ee3f03bc6814.slice/crio-d554d727f4720839fbe983009680885b94ecc61bb48978c2222259e415ab3a8f WatchSource:0}: Error finding container d554d727f4720839fbe983009680885b94ecc61bb48978c2222259e415ab3a8f: Status 404 returned error can't find the container with id d554d727f4720839fbe983009680885b94ecc61bb48978c2222259e415ab3a8f Oct 31 00:49:46.666737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.666693 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" event={"ID":"bc9a75c3-95ad-433d-8b6b-2e2817c52497","Type":"ContainerStarted","Data":"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a"} Oct 31 00:49:46.667261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.666763 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" event={"ID":"bc9a75c3-95ad-433d-8b6b-2e2817c52497","Type":"ContainerStarted","Data":"35d06f71b3885da20d49a9f6b2f4083773ddf078e3e8e901a03ff7bd3ef0306c"} Oct 31 00:49:46.667261 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.666810 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:46.668335 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.668306 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" event={"ID":"a872f8df-b5a6-4038-b55f-866b49962273","Type":"ContainerStarted","Data":"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638"} Oct 31 00:49:46.668475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.668365 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:46.668475 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.668392 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" event={"ID":"a872f8df-b5a6-4038-b55f-866b49962273","Type":"ContainerStarted","Data":"97e35f65b0bbfdd0498c93f1e6ebc3a6dfae958f06dd84a07f09a3696a81d295"} Oct 31 00:49:46.669526 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.669502 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" event={"ID":"abef928e-c3b4-491e-9bb0-ee3f03bc6814","Type":"ContainerStarted","Data":"d554d727f4720839fbe983009680885b94ecc61bb48978c2222259e415ab3a8f"} Oct 31 00:49:46.687451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.687405 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" podStartSLOduration=1.687393001 podStartE2EDuration="1.687393001s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:46.685659627 +0000 UTC m=+3308.718755699" watchObservedRunningTime="2026-10-31 00:49:46.687393001 +0000 UTC m=+3308.720489071" Oct 31 00:49:46.708973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:46.708916 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" podStartSLOduration=1.708901745 podStartE2EDuration="1.708901745s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:46.706806614 +0000 UTC m=+3308.739902688" watchObservedRunningTime="2026-10-31 00:49:46.708901745 +0000 UTC m=+3308.741997817" Oct 31 00:49:47.676434 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:47.676400 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" event={"ID":"abef928e-c3b4-491e-9bb0-ee3f03bc6814","Type":"ContainerStarted","Data":"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b"} Oct 31 00:49:47.676926 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:47.676547 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:47.708622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:47.708560 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" podStartSLOduration=2.27962536 podStartE2EDuration="2.708540577s" totalImagesPullingTime="428.915217ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:49:45 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:49:47.705740031 +0000 UTC m=+3309.738836101" watchObservedRunningTime="2026-10-31 00:49:47.708540577 +0000 UTC m=+3309.741636651" Oct 31 00:49:49.324965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:49.324923 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:49:49.325555 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:49.325132 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" podUID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" containerName="maas-api" containerID="cri-o://36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af" gracePeriod=30 Oct 31 00:49:50.621898 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.621873 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:49:50.692462 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.692426 6234 generic.go:356] "Generic (PLEG): container finished" podID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" containerID="36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af" exitCode=0 Oct 31 00:49:50.692670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.692478 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" event={"ID":"20fc4c02-e14c-42c4-bad5-8858afedc5d5","Type":"ContainerDied","Data":"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af"} Oct 31 00:49:50.692670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.692490 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" Oct 31 00:49:50.692670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.692516 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp" event={"ID":"20fc4c02-e14c-42c4-bad5-8858afedc5d5","Type":"ContainerDied","Data":"d800ac8a624c1d9ea0a47e38f6cac96e200f811eaa0530293538a5a4b8415780"} Oct 31 00:49:50.692670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.692540 6234 scope.go:122] "RemoveContainer" containerID="36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af" Oct 31 00:49:50.702682 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.702660 6234 scope.go:122] "RemoveContainer" containerID="36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af" Oct 31 00:49:50.702975 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:49:50.702956 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af\": container with ID starting with 36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af not found: ID does not exist" containerID="36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af" Oct 31 00:49:50.703046 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.702985 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af"} err="failed to get container status \"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af\": rpc error: code = NotFound desc = could not find container \"36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af\": container with ID starting with 36bb457e66ea3472d3872e0100a50aad6ccbc4821b4e21a2d05b4ab7931052af not found: ID does not exist" Oct 31 00:49:50.719320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.719295 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\") pod \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " Oct 31 00:49:50.719450 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.719414 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\") pod \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " Oct 31 00:49:50.719524 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.719464 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\" (UniqueName: \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\") pod \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " Oct 31 00:49:50.719618 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.719604 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\") pod \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\" (UID: \"20fc4c02-e14c-42c4-bad5-8858afedc5d5\") " Oct 31 00:49:50.720014 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.719990 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca" pod "20fc4c02-e14c-42c4-bad5-8858afedc5d5" (UID: "20fc4c02-e14c-42c4-bad5-8858afedc5d5"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:49:50.721632 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.721607 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls" pod "20fc4c02-e14c-42c4-bad5-8858afedc5d5" (UID: "20fc4c02-e14c-42c4-bad5-8858afedc5d5"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:50.721713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.721641 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs" pod "20fc4c02-e14c-42c4-bad5-8858afedc5d5" (UID: "20fc4c02-e14c-42c4-bad5-8858afedc5d5"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:49:50.721713 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.721662 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf" pod "20fc4c02-e14c-42c4-bad5-8858afedc5d5" (UID: "20fc4c02-e14c-42c4-bad5-8858afedc5d5"). InnerVolumeSpecName "kube-api-access-lk7cf". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:49:50.821377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.821341 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/20fc4c02-e14c-42c4-bad5-8858afedc5d5-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.821377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.821373 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.821377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.821383 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/20fc4c02-e14c-42c4-bad5-8858afedc5d5-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:50.821651 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:50.821393 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lk7cf\" (UniqueName: \"kubernetes.io/projected/20fc4c02-e14c-42c4-bad5-8858afedc5d5-kube-api-access-lk7cf\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:49:51.017346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:51.017311 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:49:51.021031 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:51.021005 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-6676c7f7bc-7f6dp"] Oct 31 00:49:52.080892 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:52.080857 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" path="/var/lib/kubelet/pods/20fc4c02-e14c-42c4-bad5-8858afedc5d5/volumes" Oct 31 00:49:53.686427 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:53.686396 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:49:57.680107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:57.680072 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:49:57.680559 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:57.680241 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:49:58.734616 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.734583 6234 generic.go:356] "Generic (PLEG): container finished" podID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerID="e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568" exitCode=0 Oct 31 00:49:58.735123 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.734661 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerDied","Data":"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568"} Oct 31 00:49:58.735191 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.735123 6234 scope.go:122] "RemoveContainer" containerID="e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568" Oct 31 00:49:58.736271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.736243 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:58.736357 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.736344 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:58.736394 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:58.736380 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:49:59.659263 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:59.656934 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:59.763117 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:59.763075 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerStarted","Data":"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681"} Oct 31 00:49:59.957815 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:59.949155 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:49:59.967809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:59.966912 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:49:59.967809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:49:59.967019 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 31 00:50:00.963819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:00.963152 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:50:00.963819 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:00.963210 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 31 00:50:01.947305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:01.947259 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:50:01.947634 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:01.947600 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 31 00:50:02.948811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:02.944925 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:50:02.948811 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:02.944990 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 31 00:50:03.185216 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:03.185175 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:50:03.953257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:03.953215 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" start-of-body= Oct 31 00:50:03.953938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:03.953899 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.171:15021/healthz/ready\": dial tcp 10.134.0.171:15021: connect: connection refused" Oct 31 00:50:04.947920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:04.947875 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:50:04.948272 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:04.948240 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:50:04.949067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:04.949048 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:50:10.342636 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.342598 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:50:10.347407 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.347373 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:50:10.457297 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.457262 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:50:10.457540 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.457513 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" podUID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" containerName="maas-api" containerID="cri-o://cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b" gracePeriod=30 Oct 31 00:50:10.744739 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.744713 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:50:10.794844 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.794776 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\") pod \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " Oct 31 00:50:10.795037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.794865 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\") pod \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " Oct 31 00:50:10.795037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.794943 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\") pod \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " Oct 31 00:50:10.795037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.795002 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\" (UniqueName: \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\") pod \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\" (UID: \"abef928e-c3b4-491e-9bb0-ee3f03bc6814\") " Oct 31 00:50:10.795333 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.795307 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca" pod "abef928e-c3b4-491e-9bb0-ee3f03bc6814" (UID: "abef928e-c3b4-491e-9bb0-ee3f03bc6814"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:10.795489 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.795468 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/abef928e-c3b4-491e-9bb0-ee3f03bc6814-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.797359 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.797327 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8" pod "abef928e-c3b4-491e-9bb0-ee3f03bc6814" (UID: "abef928e-c3b4-491e-9bb0-ee3f03bc6814"). InnerVolumeSpecName "kube-api-access-z7qx8". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:10.798087 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.798052 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls" pod "abef928e-c3b4-491e-9bb0-ee3f03bc6814" (UID: "abef928e-c3b4-491e-9bb0-ee3f03bc6814"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.798184 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.798088 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs" pod "abef928e-c3b4-491e-9bb0-ee3f03bc6814" (UID: "abef928e-c3b4-491e-9bb0-ee3f03bc6814"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:50:10.830351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.830305 6234 generic.go:356] "Generic (PLEG): container finished" podID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" containerID="cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b" exitCode=0 Oct 31 00:50:10.830537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.830355 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" Oct 31 00:50:10.830537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.830379 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" event={"ID":"abef928e-c3b4-491e-9bb0-ee3f03bc6814","Type":"ContainerDied","Data":"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b"} Oct 31 00:50:10.830537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.830440 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt" event={"ID":"abef928e-c3b4-491e-9bb0-ee3f03bc6814","Type":"ContainerDied","Data":"d554d727f4720839fbe983009680885b94ecc61bb48978c2222259e415ab3a8f"} Oct 31 00:50:10.830537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.830467 6234 scope.go:122] "RemoveContainer" containerID="cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b" Oct 31 00:50:10.840502 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.840474 6234 scope.go:122] "RemoveContainer" containerID="cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b" Oct 31 00:50:10.840924 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:10.840901 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b\": container with ID starting with cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b not found: ID does not exist" containerID="cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b" Oct 31 00:50:10.841017 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.840936 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b"} err="failed to get container status \"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b\": rpc error: code = NotFound desc = could not find container \"cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b\": container with ID starting with cfc0c41f01d6f8491be5c8331ca5ce1856fc4a7ed494fdeca0928e6a5b50ab8b not found: ID does not exist" Oct 31 00:50:10.855491 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.855452 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:50:10.859099 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.859073 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-fce3e276-fc5897f7b-2f8gt"] Oct 31 00:50:10.896717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.896627 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.896717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.896661 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/abef928e-c3b4-491e-9bb0-ee3f03bc6814-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:10.896717 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:10.896678 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z7qx8\" (UniqueName: \"kubernetes.io/projected/abef928e-c3b4-491e-9bb0-ee3f03bc6814-kube-api-access-z7qx8\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:12.080901 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:12.080863 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" path="/var/lib/kubelet/pods/abef928e-c3b4-491e-9bb0-ee3f03bc6814/volumes" Oct 31 00:50:15.343597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.343554 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" podUID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" containerName="payload-processing" containerID="cri-o://448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a" gracePeriod=25 Oct 31 00:50:15.347916 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.347885 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" podUID="a872f8df-b5a6-4038-b55f-866b49962273" containerName="payload-pre-processing" containerID="cri-o://d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638" gracePeriod=25 Oct 31 00:50:15.613459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.613432 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:50:15.616606 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.616581 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:50:15.745114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745078 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\" (UniqueName: \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\") pod \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " Oct 31 00:50:15.745331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745151 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\") pod \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\" (UID: \"bc9a75c3-95ad-433d-8b6b-2e2817c52497\") " Oct 31 00:50:15.745331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745308 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\") pod \"a872f8df-b5a6-4038-b55f-866b49962273\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " Oct 31 00:50:15.745456 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745361 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\" (UniqueName: \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\") pod \"a872f8df-b5a6-4038-b55f-866b49962273\" (UID: \"a872f8df-b5a6-4038-b55f-866b49962273\") " Oct 31 00:50:15.745515 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745495 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume" pod "bc9a75c3-95ad-433d-8b6b-2e2817c52497" (UID: "bc9a75c3-95ad-433d-8b6b-2e2817c52497"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:15.745723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745693 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume" pod "a872f8df-b5a6-4038-b55f-866b49962273" (UID: "a872f8df-b5a6-4038-b55f-866b49962273"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:15.745723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.745712 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bc9a75c3-95ad-433d-8b6b-2e2817c52497-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:15.747507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.747477 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4" pod "bc9a75c3-95ad-433d-8b6b-2e2817c52497" (UID: "bc9a75c3-95ad-433d-8b6b-2e2817c52497"). InnerVolumeSpecName "kube-api-access-hdfn4". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:15.747507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.747489 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2" pod "a872f8df-b5a6-4038-b55f-866b49962273" (UID: "a872f8df-b5a6-4038-b55f-866b49962273"). InnerVolumeSpecName "kube-api-access-564w2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:15.847026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.846938 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a872f8df-b5a6-4038-b55f-866b49962273-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:15.847026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.846971 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-564w2\" (UniqueName: \"kubernetes.io/projected/a872f8df-b5a6-4038-b55f-866b49962273-kube-api-access-564w2\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:15.847026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.846982 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hdfn4\" (UniqueName: \"kubernetes.io/projected/bc9a75c3-95ad-433d-8b6b-2e2817c52497-kube-api-access-hdfn4\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:15.856988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.856956 6234 generic.go:356] "Generic (PLEG): container finished" podID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" containerID="448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a" exitCode=0 Oct 31 00:50:15.857150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.857035 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" Oct 31 00:50:15.857150 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.857036 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" event={"ID":"bc9a75c3-95ad-433d-8b6b-2e2817c52497","Type":"ContainerDied","Data":"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a"} Oct 31 00:50:15.857260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.857207 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk" event={"ID":"bc9a75c3-95ad-433d-8b6b-2e2817c52497","Type":"ContainerDied","Data":"35d06f71b3885da20d49a9f6b2f4083773ddf078e3e8e901a03ff7bd3ef0306c"} Oct 31 00:50:15.857260 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.857233 6234 scope.go:122] "RemoveContainer" containerID="448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a" Oct 31 00:50:15.858338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.858316 6234 generic.go:356] "Generic (PLEG): container finished" podID="a872f8df-b5a6-4038-b55f-866b49962273" containerID="d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638" exitCode=0 Oct 31 00:50:15.858413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.858366 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" Oct 31 00:50:15.858413 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.858397 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" event={"ID":"a872f8df-b5a6-4038-b55f-866b49962273","Type":"ContainerDied","Data":"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638"} Oct 31 00:50:15.858503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.858437 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml" event={"ID":"a872f8df-b5a6-4038-b55f-866b49962273","Type":"ContainerDied","Data":"97e35f65b0bbfdd0498c93f1e6ebc3a6dfae958f06dd84a07f09a3696a81d295"} Oct 31 00:50:15.866528 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.866436 6234 scope.go:122] "RemoveContainer" containerID="448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a" Oct 31 00:50:15.866766 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:15.866737 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a\": container with ID starting with 448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a not found: ID does not exist" containerID="448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a" Oct 31 00:50:15.866929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.866774 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a"} err="failed to get container status \"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a\": rpc error: code = NotFound desc = could not find container \"448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a\": container with ID starting with 448bd16d5ef3af90aba2e4d302a7097fb2313e4ffb62daff2633d316ab32368a not found: ID does not exist" Oct 31 00:50:15.866929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.866807 6234 scope.go:122] "RemoveContainer" containerID="d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638" Oct 31 00:50:15.875110 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.875090 6234 scope.go:122] "RemoveContainer" containerID="d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638" Oct 31 00:50:15.875397 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:15.875377 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638\": container with ID starting with d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638 not found: ID does not exist" containerID="d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638" Oct 31 00:50:15.875473 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.875406 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638"} err="failed to get container status \"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638\": rpc error: code = NotFound desc = could not find container \"d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638\": container with ID starting with d43b01fb66bda62a7a8b1762715186689712e50b8b48b57fe0df365232acd638 not found: ID does not exist" Oct 31 00:50:15.883537 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.883514 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:50:15.891212 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.891188 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-fce3e276-54fd866f75-6znml"] Oct 31 00:50:15.903955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.903929 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:50:15.909546 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:15.909519 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-fce3e276-f4f68c477-xrfbk"] Oct 31 00:50:16.080592 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:16.080549 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a872f8df-b5a6-4038-b55f-866b49962273" path="/var/lib/kubelet/pods/a872f8df-b5a6-4038-b55f-866b49962273/volumes" Oct 31 00:50:16.081056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:16.081035 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" path="/var/lib/kubelet/pods/bc9a75c3-95ad-433d-8b6b-2e2817c52497/volumes" Oct 31 00:50:16.616024 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:16.615993 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:50:16.616626 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:16.616245 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" containerID="cri-o://436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681" gracePeriod=30 Oct 31 00:50:18.415157 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415067 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415538 6234 cpu_manager.go:414] "containerMap: removing container" podUID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" containerName="maas-api" Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415552 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" containerName="maas-api" Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415570 6234 cpu_manager.go:414] "containerMap: removing container" podUID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" containerName="maas-api" Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415575 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" containerName="maas-api" Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415585 6234 cpu_manager.go:414] "containerMap: removing container" podUID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" containerName="payload-processing" Oct 31 00:50:18.415596 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415593 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" containerName="payload-processing" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415605 6234 cpu_manager.go:414] "containerMap: removing container" podUID="a872f8df-b5a6-4038-b55f-866b49962273" containerName="payload-pre-processing" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415610 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a872f8df-b5a6-4038-b55f-866b49962273" containerName="payload-pre-processing" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415695 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="abef928e-c3b4-491e-9bb0-ee3f03bc6814" containerName="maas-api" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415707 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="20fc4c02-e14c-42c4-bad5-8858afedc5d5" containerName="maas-api" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415713 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="bc9a75c3-95ad-433d-8b6b-2e2817c52497" containerName="payload-processing" Oct 31 00:50:18.415850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.415720 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="a872f8df-b5a6-4038-b55f-866b49962273" containerName="payload-pre-processing" Oct 31 00:50:18.420697 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.420670 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.427174 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.427141 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-36b16a-openshift-default-dockercfg-897hk\"" Oct 31 00:50:18.437999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.437975 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:50:18.474954 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.474916 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97zmt\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.474962 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475053 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475140 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475113 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475187 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475313 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475362 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475432 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475422 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.475519 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.475454 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.576895 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.576855 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.576932 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.576964 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577005 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577079 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577041 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577090 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97zmt\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577110 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577133 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577295 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577168 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577350 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577437 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577480 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577746 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.577902 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.577887 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.579509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.579489 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.579720 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.579700 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.591070 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.591035 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.591196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.591171 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-97zmt\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\") pod \"e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:18.734655 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:18.734619 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:19.020581 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.020555 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:50:19.022376 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:50:19.022347 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8139042a_62c7_4354_8c2a_956c24fd5e01.slice/crio-ba4c282b0cffcfa4958eb7317032160a5e0f9806c8640b6c4ae41e1782aa9ce6 WatchSource:0}: Error finding container ba4c282b0cffcfa4958eb7317032160a5e0f9806c8640b6c4ae41e1782aa9ce6: Status 404 returned error can't find the container with id ba4c282b0cffcfa4958eb7317032160a5e0f9806c8640b6c4ae41e1782aa9ce6 Oct 31 00:50:19.024689 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.024653 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:19.024817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.024735 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:19.024817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.024799 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:19.881512 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.881474 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerStarted","Data":"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8"} Oct 31 00:50:19.882020 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.881530 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerStarted","Data":"ba4c282b0cffcfa4958eb7317032160a5e0f9806c8640b6c4ae41e1782aa9ce6"} Oct 31 00:50:19.909686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.909638 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podStartSLOduration=1.9096241859999998 podStartE2EDuration="1.909624186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:19.908286424 +0000 UTC m=+3341.941382494" watchObservedRunningTime="2026-10-31 00:50:19.909624186 +0000 UTC m=+3341.942720255" Oct 31 00:50:19.950732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.950695 6234 patch_prober.go:29] interesting pod/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused Oct 31 00:50:19.950931 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:19.950758 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503" Oct 31 00:50:20.735091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:20.735048 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:20.740052 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:20.740025 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:20.886573 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:20.886536 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:20.887481 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:20.887462 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:21.873387 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.873360 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:50:21.892722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.892640 6234 generic.go:356] "Generic (PLEG): container finished" podID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerID="436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681" exitCode=0 Oct 31 00:50:21.892722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.892690 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" Oct 31 00:50:21.892722 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.892712 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerDied","Data":"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681"} Oct 31 00:50:21.893227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.892763 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf" event={"ID":"9826ba3c-5b50-47c2-b3b3-85217f011406","Type":"ContainerDied","Data":"e4f6fedf43d4536c4b09625cfbe93dcbb165635bb1b15d2a01d94d2c76ee702c"} Oct 31 00:50:21.893227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.892799 6234 scope.go:122] "RemoveContainer" containerID="436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681" Oct 31 00:50:21.902880 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.902769 6234 scope.go:122] "RemoveContainer" containerID="e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568" Oct 31 00:50:21.916719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.916691 6234 scope.go:122] "RemoveContainer" containerID="436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681" Oct 31 00:50:21.917272 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:21.917247 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681\": container with ID starting with 436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681 not found: ID does not exist" containerID="436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681" Oct 31 00:50:21.917391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.917280 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681"} err="failed to get container status \"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681\": rpc error: code = NotFound desc = could not find container \"436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681\": container with ID starting with 436210e4b3d55ee03fbe2970d6aa99b4c8ca07a2b219be01120632d89473f681 not found: ID does not exist" Oct 31 00:50:21.917391 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.917300 6234 scope.go:122] "RemoveContainer" containerID="e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568" Oct 31 00:50:21.917662 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:21.917643 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568\": container with ID starting with e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568 not found: ID does not exist" containerID="e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568" Oct 31 00:50:21.917736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:21.917670 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568"} err="failed to get container status \"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568\": rpc error: code = NotFound desc = could not find container \"e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568\": container with ID starting with e98bb44ce71e7d9b920888e731d2d9e94530914f4e5d52e35b20cc8106cda568 not found: ID does not exist" Oct 31 00:50:22.012686 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012648 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.012929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012707 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.012929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012745 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.012929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012833 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.012929 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012866 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012942 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012975 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013014 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.012992 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013087 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\") pod \"9826ba3c-5b50-47c2-b3b3-85217f011406\" (UID: \"9826ba3c-5b50-47c2-b3b3-85217f011406\") " Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013207 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:22.013292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013284 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:22.013837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013335 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:50:22.013837 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013532 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:22.013991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013969 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.014057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.013999 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.014057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.014015 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9826ba3c-5b50-47c2-b3b3-85217f011406-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.014057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.014030 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.014057 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.014043 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.015554 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.015521 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "kube-api-access-c698s". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:22.015999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.015975 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:50:22.015999 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.015983 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:50:22.016305 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.016290 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy" pod "9826ba3c-5b50-47c2-b3b3-85217f011406" (UID: "9826ba3c-5b50-47c2-b3b3-85217f011406"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:50:22.115138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.115103 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.115138 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.115134 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c698s\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-kube-api-access-c698s\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.115352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.115146 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.115352 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.115162 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9826ba3c-5b50-47c2-b3b3-85217f011406-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:50:22.215478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.215445 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:50:22.226282 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:22.226249 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-fce3e276-openshift-default-5bc996c695-xs9bf"] Oct 31 00:50:23.331499 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.331460 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:50:23.332605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.332081 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.332605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.332104 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.332605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.332268 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.332605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.332289 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.336408 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.336386 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.348985 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.348957 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:50:23.385114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.385085 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-authcfg-http-36b16a-serving-cert\"" Oct 31 00:50:23.428503 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.428466 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.428690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.428508 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9pjqh\" (UniqueName: \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.428690 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.428657 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.428840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.428717 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.530185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.530148 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.530478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.530213 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.530478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.530241 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9pjqh\" (UniqueName: \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.530478 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.530302 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.530991 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.530964 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.533351 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.533316 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.533538 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.533520 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.533911 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.533889 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:50:23.534373 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.534358 6234 cpu_manager.go:414] "containerMap: removing container" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.534460 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.534377 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" containerName="istio-proxy" Oct 31 00:50:23.537671 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.537653 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.546840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.546822 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-authcfg-http-36b16a-dockercfg-rp6qp\"" Oct 31 00:50:23.548850 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.548826 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:50:23.550048 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.550027 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-authcfg-http-36b16a\"" Oct 31 00:50:23.558872 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.558850 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9pjqh\" (UniqueName: \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\") pod \"maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.572162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.572142 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:50:23.575792 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.575763 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.585993 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.585967 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:50:23.632852 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.632810 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.633056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.632896 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.633108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.633079 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4wnf2\" (UniqueName: \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.633196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.633177 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9wnr\" (UniqueName: \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.650767 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.650724 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.734254 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.734309 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.734354 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4wnf2\" (UniqueName: \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.734391 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9wnr\" (UniqueName: \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.735544 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.735905 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.735600 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.748327 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.748299 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9wnr\" (UniqueName: \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\") pod \"payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.760909 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.760881 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4wnf2\" (UniqueName: \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\") pod \"payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.863585 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.863508 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:23.890050 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.890022 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:23.979887 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:23.979804 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:50:24.083917 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.082734 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9826ba3c-5b50-47c2-b3b3-85217f011406" path="/var/lib/kubelet/pods/9826ba3c-5b50-47c2-b3b3-85217f011406/volumes" Oct 31 00:50:24.141152 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.141119 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:50:24.143050 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:50:24.143018 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3c4caced_6f6c_44fb_8907_9c71d949240b.slice/crio-5a5da56afdbd7d4e8a5ca3a8339dc8086bc83a725136ef47684da023eb13eccb WatchSource:0}: Error finding container 5a5da56afdbd7d4e8a5ca3a8339dc8086bc83a725136ef47684da023eb13eccb: Status 404 returned error can't find the container with id 5a5da56afdbd7d4e8a5ca3a8339dc8086bc83a725136ef47684da023eb13eccb Oct 31 00:50:24.203171 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.203142 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:50:24.205170 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:50:24.205146 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4cb5df6_9025_4b06_af6b_1c5fb50d6b8c.slice/crio-8388d640b3744268e5fccc88ae3bdb09a7f3e7a28bc67e0833c1143a3883c0f4 WatchSource:0}: Error finding container 8388d640b3744268e5fccc88ae3bdb09a7f3e7a28bc67e0833c1143a3883c0f4: Status 404 returned error can't find the container with id 8388d640b3744268e5fccc88ae3bdb09a7f3e7a28bc67e0833c1143a3883c0f4 Oct 31 00:50:24.910723 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.910679 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" event={"ID":"3c4caced-6f6c-44fb-8907-9c71d949240b","Type":"ContainerStarted","Data":"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02"} Oct 31 00:50:24.911235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.910743 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" event={"ID":"3c4caced-6f6c-44fb-8907-9c71d949240b","Type":"ContainerStarted","Data":"5a5da56afdbd7d4e8a5ca3a8339dc8086bc83a725136ef47684da023eb13eccb"} Oct 31 00:50:24.911235 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.910912 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:24.912170 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.912148 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" event={"ID":"3513c23e-0879-4525-91dd-62acc9c74136","Type":"ContainerStarted","Data":"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306"} Oct 31 00:50:24.912236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.912196 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:24.912236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.912218 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" event={"ID":"3513c23e-0879-4525-91dd-62acc9c74136","Type":"ContainerStarted","Data":"4ad5c0a075059eecae5898aa40676b9ac9fa072ad124bd1f4f42f5621e705f85"} Oct 31 00:50:24.913676 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.913651 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" event={"ID":"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c","Type":"ContainerStarted","Data":"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df"} Oct 31 00:50:24.913828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.913686 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" event={"ID":"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c","Type":"ContainerStarted","Data":"8388d640b3744268e5fccc88ae3bdb09a7f3e7a28bc67e0833c1143a3883c0f4"} Oct 31 00:50:24.913828 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.913732 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:24.932300 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.932255 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" podStartSLOduration=1.9322320990000001 podStartE2EDuration="1.932232099s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:24.931172682 +0000 UTC m=+3346.964268750" watchObservedRunningTime="2026-10-31 00:50:24.932232099 +0000 UTC m=+3346.965328165" Oct 31 00:50:24.952204 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.952154 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" podStartSLOduration=1.952142064 podStartE2EDuration="1.952142064s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:24.951734493 +0000 UTC m=+3346.984830564" watchObservedRunningTime="2026-10-31 00:50:24.952142064 +0000 UTC m=+3346.985238131" Oct 31 00:50:24.975897 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:24.975854 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" podStartSLOduration=1.5450116120000001 podStartE2EDuration="1.975841005s" totalImagesPullingTime="430.829393ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:50:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:50:24.974640652 +0000 UTC m=+3347.007736715" watchObservedRunningTime="2026-10-31 00:50:24.975841005 +0000 UTC m=+3347.008937071" Oct 31 00:50:30.926039 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:30.926009 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:50:33.161406 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:50:33.161305 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:50:35.921970 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:35.921932 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:50:35.922451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:35.922008 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:50:36.036955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:36.034006 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": read tcp 10.134.0.2:33708->10.134.0.175:15021: read: connection reset by peer" start-of-body= Oct 31 00:50:36.036955 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:36.034084 6234 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": read tcp 10.134.0.2:33708->10.134.0.175:15021: read: connection reset by peer" Oct 31 00:50:37.003601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.003566 6234 generic.go:356] "Generic (PLEG): container finished" podID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerID="679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8" exitCode=0 Oct 31 00:50:37.004019 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.003642 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerDied","Data":"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8"} Oct 31 00:50:37.004106 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.004091 6234 scope.go:122] "RemoveContainer" containerID="679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8" Oct 31 00:50:37.005405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.005379 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:37.005470 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.005448 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:37.005505 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:37.005479 6234 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892000Ki","pods":"250"} Oct 31 00:50:38.010414 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:38.010378 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerStarted","Data":"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32"} Oct 31 00:50:38.734732 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:38.734698 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:38.735016 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:38.734996 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 00:50:38.735066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:38.735041 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 00:50:39.735621 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:39.735576 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 00:50:39.736092 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:39.735646 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 00:50:40.735429 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:40.735389 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 00:50:40.735598 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:40.735445 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 00:50:41.735711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:41.735668 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 00:50:41.736182 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:41.735728 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 00:50:42.735268 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:42.735221 6234 patch_prober.go:29] interesting pod/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" start-of-body= Oct 31 00:50:42.735464 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:42.735286 6234 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.175:15021/healthz/ready\": dial tcp 10.134.0.175:15021: connect: connection refused" Oct 31 00:50:43.739590 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:43.739558 6234 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:43.740107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:43.739989 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:50:43.740714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:50:43.740697 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:51:03.161182 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:51:03.161133 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:51:33.162305 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:51:33.162268 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:52:03.162445 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:03.162357 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:52:07.852628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:07.852594 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:52:07.866693 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:07.866664 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:52:12.853332 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:12.853284 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" podUID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" containerName="payload-processing" containerID="cri-o://2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df" gracePeriod=25 Oct 31 00:52:12.867468 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:12.867359 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" podUID="3c4caced-6f6c-44fb-8907-9c71d949240b" containerName="payload-pre-processing" containerID="cri-o://1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02" gracePeriod=25 Oct 31 00:52:12.877422 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:12.877397 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:52:12.877642 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:12.877613 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" podUID="3513c23e-0879-4525-91dd-62acc9c74136" containerName="maas-api" containerID="cri-o://82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306" gracePeriod=30 Oct 31 00:52:13.192531 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.192499 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:52:13.204091 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.204030 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:52:13.243183 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.243153 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:52:13.249122 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249092 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\") pod \"3c4caced-6f6c-44fb-8907-9c71d949240b\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " Oct 31 00:52:13.249275 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249146 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\" (UniqueName: \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\") pod \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " Oct 31 00:52:13.249346 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249301 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\") pod \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\" (UID: \"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c\") " Oct 31 00:52:13.249411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249381 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\" (UniqueName: \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\") pod \"3c4caced-6f6c-44fb-8907-9c71d949240b\" (UID: \"3c4caced-6f6c-44fb-8907-9c71d949240b\") " Oct 31 00:52:13.249628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249528 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume" pod "3c4caced-6f6c-44fb-8907-9c71d949240b" (UID: "3c4caced-6f6c-44fb-8907-9c71d949240b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:13.249737 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249712 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume" pod "d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" (UID: "d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:13.249840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.249821 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3c4caced-6f6c-44fb-8907-9c71d949240b-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.253094 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.253073 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr" pod "d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" (UID: "d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c"). InnerVolumeSpecName "kube-api-access-m9wnr". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:13.254650 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.254627 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2" pod "3c4caced-6f6c-44fb-8907-9c71d949240b" (UID: "3c4caced-6f6c-44fb-8907-9c71d949240b"). InnerVolumeSpecName "kube-api-access-4wnf2". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:13.350608 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.350571 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\") pod \"3513c23e-0879-4525-91dd-62acc9c74136\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " Oct 31 00:52:13.350838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.350661 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\") pod \"3513c23e-0879-4525-91dd-62acc9c74136\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " Oct 31 00:52:13.350838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.350734 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\") pod \"3513c23e-0879-4525-91dd-62acc9c74136\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " Oct 31 00:52:13.350990 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.350858 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\" (UniqueName: \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\") pod \"3513c23e-0879-4525-91dd-62acc9c74136\" (UID: \"3513c23e-0879-4525-91dd-62acc9c74136\") " Oct 31 00:52:13.351072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.351050 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca" pod "3513c23e-0879-4525-91dd-62acc9c74136" (UID: "3513c23e-0879-4525-91dd-62acc9c74136"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:13.351236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.351212 6234 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-plugins-config-volume\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.351236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.351230 6234 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/3513c23e-0879-4525-91dd-62acc9c74136-openshift-service-ca\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.351236 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.351240 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4wnf2\" (UniqueName: \"kubernetes.io/projected/3c4caced-6f6c-44fb-8907-9c71d949240b-kube-api-access-4wnf2\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.351649 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.351249 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9wnr\" (UniqueName: \"kubernetes.io/projected/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c-kube-api-access-m9wnr\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.352971 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.352949 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls" pod "3513c23e-0879-4525-91dd-62acc9c74136" (UID: "3513c23e-0879-4525-91dd-62acc9c74136"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:13.353067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.353007 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs" pod "3513c23e-0879-4525-91dd-62acc9c74136" (UID: "3513c23e-0879-4525-91dd-62acc9c74136"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue "" Oct 31 00:52:13.353067 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.353009 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh" pod "3513c23e-0879-4525-91dd-62acc9c74136" (UID: "3513c23e-0879-4525-91dd-62acc9c74136"). InnerVolumeSpecName "kube-api-access-9pjqh". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:13.452149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.452118 6234 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-maas-api-tls\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.452149 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.452149 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9pjqh\" (UniqueName: \"kubernetes.io/projected/3513c23e-0879-4525-91dd-62acc9c74136-kube-api-access-9pjqh\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.452354 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.452159 6234 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3513c23e-0879-4525-91dd-62acc9c74136-metrics-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:13.528299 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.528258 6234 generic.go:356] "Generic (PLEG): container finished" podID="3513c23e-0879-4525-91dd-62acc9c74136" containerID="82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306" exitCode=0 Oct 31 00:52:13.528497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.528304 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" Oct 31 00:52:13.528497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.528344 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" event={"ID":"3513c23e-0879-4525-91dd-62acc9c74136","Type":"ContainerDied","Data":"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306"} Oct 31 00:52:13.528497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.528405 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9" event={"ID":"3513c23e-0879-4525-91dd-62acc9c74136","Type":"ContainerDied","Data":"4ad5c0a075059eecae5898aa40676b9ac9fa072ad124bd1f4f42f5621e705f85"} Oct 31 00:52:13.528497 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.528433 6234 scope.go:122] "RemoveContainer" containerID="82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306" Oct 31 00:52:13.529933 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.529910 6234 generic.go:356] "Generic (PLEG): container finished" podID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" containerID="2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df" exitCode=0 Oct 31 00:52:13.530056 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.529960 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" Oct 31 00:52:13.530114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.530078 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" event={"ID":"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c","Type":"ContainerDied","Data":"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df"} Oct 31 00:52:13.530167 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.530118 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76" event={"ID":"d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c","Type":"ContainerDied","Data":"8388d640b3744268e5fccc88ae3bdb09a7f3e7a28bc67e0833c1143a3883c0f4"} Oct 31 00:52:13.534727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.531624 6234 generic.go:356] "Generic (PLEG): container finished" podID="3c4caced-6f6c-44fb-8907-9c71d949240b" containerID="1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02" exitCode=0 Oct 31 00:52:13.534727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.531709 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" event={"ID":"3c4caced-6f6c-44fb-8907-9c71d949240b","Type":"ContainerDied","Data":"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02"} Oct 31 00:52:13.534727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.531748 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" event={"ID":"3c4caced-6f6c-44fb-8907-9c71d949240b","Type":"ContainerDied","Data":"5a5da56afdbd7d4e8a5ca3a8339dc8086bc83a725136ef47684da023eb13eccb"} Oct 31 00:52:13.534727 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.531829 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq" Oct 31 00:52:13.541904 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.541877 6234 scope.go:122] "RemoveContainer" containerID="82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306" Oct 31 00:52:13.542280 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:13.542209 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306\": container with ID starting with 82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306 not found: ID does not exist" containerID="82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306" Oct 31 00:52:13.542280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.542240 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306"} err="failed to get container status \"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306\": rpc error: code = NotFound desc = could not find container \"82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306\": container with ID starting with 82206eaecf30f97734c8306c29443e4e2220df91be633b9556b576211a9ef306 not found: ID does not exist" Oct 31 00:52:13.542280 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.542257 6234 scope.go:122] "RemoveContainer" containerID="2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df" Oct 31 00:52:13.555751 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.555731 6234 scope.go:122] "RemoveContainer" containerID="2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df" Oct 31 00:52:13.556133 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:13.556100 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df\": container with ID starting with 2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df not found: ID does not exist" containerID="2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df" Oct 31 00:52:13.556226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.556143 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df"} err="failed to get container status \"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df\": rpc error: code = NotFound desc = could not find container \"2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df\": container with ID starting with 2dd27025b49a4cd5b8e35cfdfa7715b4926f840c31315c77b89b8fa2e85b70df not found: ID does not exist" Oct 31 00:52:13.556226 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.556167 6234 scope.go:122] "RemoveContainer" containerID="1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02" Oct 31 00:52:13.561026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.561005 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:52:13.564988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.564964 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-authcfg-http-36b16a-84455b8fc-6fjk9"] Oct 31 00:52:13.566762 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.566745 6234 scope.go:122] "RemoveContainer" containerID="1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02" Oct 31 00:52:13.567099 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:13.567075 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02\": container with ID starting with 1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02 not found: ID does not exist" containerID="1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02" Oct 31 00:52:13.567177 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.567106 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02"} err="failed to get container status \"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02\": rpc error: code = NotFound desc = could not find container \"1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02\": container with ID starting with 1199b8929fb3005780542a8e382f3c7b1b6375554873df7579385a2fdabcdc02 not found: ID does not exist" Oct 31 00:52:13.578670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.578649 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:52:13.583181 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.583160 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-authcfg-http-36b16a-c57f445bf-4xh76"] Oct 31 00:52:13.596876 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.596849 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:52:13.600953 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:13.600932 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-authcfg-http-36b16a-7b6c8dd56c-clzbq"] Oct 31 00:52:14.080603 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:14.080570 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3513c23e-0879-4525-91dd-62acc9c74136" path="/var/lib/kubelet/pods/3513c23e-0879-4525-91dd-62acc9c74136/volumes" Oct 31 00:52:14.081047 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:14.080974 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c4caced-6f6c-44fb-8907-9c71d949240b" path="/var/lib/kubelet/pods/3c4caced-6f6c-44fb-8907-9c71d949240b/volumes" Oct 31 00:52:14.081309 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:14.081295 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" path="/var/lib/kubelet/pods/d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c/volumes" Oct 31 00:52:20.092809 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:20.092749 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:52:20.093285 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:20.093020 6234 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" containerID="cri-o://51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32" gracePeriod=30 Oct 31 00:52:25.346225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.346198 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:52:25.519331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519298 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519358 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519384 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519413 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519517 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519453 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519561 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519592 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519627 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519638 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519647 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.519736 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519707 6234 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\") pod \"8139042a-62c7-4354-8c2a-956c24fd5e01\" (UID: \"8139042a-62c7-4354-8c2a-956c24fd5e01\") " Oct 31 00:52:25.520121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.519983 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:25.520121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520042 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue "" Oct 31 00:52:25.520121 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520055 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:25.520271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520221 6234 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8139042a-62c7-4354-8c2a-956c24fd5e01-istiod-ca-cert\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.520271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520244 6234 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-credential-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.520271 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520259 6234 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-certs\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.520405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520273 6234 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-data\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.520405 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.520287 6234 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-workload-socket\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.521965 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.521942 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:25.522120 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.522066 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue "" Oct 31 00:52:25.522402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.522379 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "kube-api-access-97zmt". PluginName "kubernetes.io/projected", VolumeGIDValue "" Oct 31 00:52:25.522402 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.522388 6234 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo" pod "8139042a-62c7-4354-8c2a-956c24fd5e01" (UID: "8139042a-62c7-4354-8c2a-956c24fd5e01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue "" Oct 31 00:52:25.621338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.621306 6234 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-token\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.621338 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.621333 6234 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-envoy\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.621529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.621345 6234 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97zmt\" (UniqueName: \"kubernetes.io/projected/8139042a-62c7-4354-8c2a-956c24fd5e01-kube-api-access-97zmt\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.621529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.621355 6234 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8139042a-62c7-4354-8c2a-956c24fd5e01-istio-podinfo\") on node \"ip-10-0-130-152.us-east-2.compute.internal\" DevicePath \"\"" Oct 31 00:52:25.623714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.623688 6234 generic.go:356] "Generic (PLEG): container finished" podID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerID="51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32" exitCode=0 Oct 31 00:52:25.623849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.623734 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerDied","Data":"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32"} Oct 31 00:52:25.623849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.623742 6234 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" Oct 31 00:52:25.623849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.623778 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc" event={"ID":"8139042a-62c7-4354-8c2a-956c24fd5e01","Type":"ContainerDied","Data":"ba4c282b0cffcfa4958eb7317032160a5e0f9806c8640b6c4ae41e1782aa9ce6"} Oct 31 00:52:25.623849 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.623822 6234 scope.go:122] "RemoveContainer" containerID="51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32" Oct 31 00:52:25.633196 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.633175 6234 scope.go:122] "RemoveContainer" containerID="679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8" Oct 31 00:52:25.641227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.641209 6234 scope.go:122] "RemoveContainer" containerID="51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32" Oct 31 00:52:25.641494 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:25.641473 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32\": container with ID starting with 51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32 not found: ID does not exist" containerID="51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32" Oct 31 00:52:25.641549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.641505 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32"} err="failed to get container status \"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32\": rpc error: code = NotFound desc = could not find container \"51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32\": container with ID starting with 51f082d59b22f29af98a5ee8d118b81f255f6e5cf6eebea93e57d893ee29fc32 not found: ID does not exist" Oct 31 00:52:25.641549 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.641523 6234 scope.go:122] "RemoveContainer" containerID="679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8" Oct 31 00:52:25.641830 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:25.641810 6234 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8\": container with ID starting with 679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8 not found: ID does not exist" containerID="679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8" Oct 31 00:52:25.641981 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.641833 6234 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8"} err="failed to get container status \"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8\": rpc error: code = NotFound desc = could not find container \"679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8\": container with ID starting with 679875f17bdaecf2294b64c9f4d8ba80122ad644dc72260f36a9b08d7d0af3e8 not found: ID does not exist" Oct 31 00:52:25.647771 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.647749 6234 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:52:25.652097 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:25.652078 6234 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-36b16a-openshift-default-58fc77fbf7-kjqsc"] Oct 31 00:52:26.079988 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:52:26.079957 6234 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" path="/var/lib/kubelet/pods/8139042a-62c7-4354-8c2a-956c24fd5e01/volumes" Oct 31 00:52:33.161226 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:52:33.161187 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:53:03.161159 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:53:03.161115 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:53:22.885555 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:22.885482 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jr6nm_130cd323-4efe-41e8-878a-d3b25ea31917/manager/0.log" Oct 31 00:53:23.584714 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:23.584682 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:53:23.814417 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:23.814383 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sxsw6_b9dad48a-5c54-4db3-a98e-581219b1046c/server/0.log" Oct 31 00:53:24.010831 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:24.010802 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mpwvn_6124af8d-fa59-479c-b6c0-561a759e7e71/manager/0.log" Oct 31 00:53:24.258711 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:24.258682 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4w2nk_5b700d0c-b689-4767-9b25-aa14f4131036/proxy/0.log" Oct 31 00:53:25.515625 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:25.515597 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8598cb9b7-rptl2_263c44c2-b7db-4633-b0ee-653c5a35d1af/maas-api/0.log" Oct 31 00:53:27.030098 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:27.030066 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ngrsz_4a721706-8e9c-454a-81d0-19e4d564eebf/postgres/0.log" Oct 31 00:53:31.350668 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:31.350635 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qzxwc_de47760d-2eff-44ce-8aa5-9d82b9842a65/discovery/0.log" Oct 31 00:53:31.576576 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:31.576547 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-m2xt8_6521afb3-77b6-4c63-a867-e15f132b1acf/kube-auth-proxy/0.log" Oct 31 00:53:32.794938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:32.794909 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/1.log" Oct 31 00:53:32.990108 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:32.990079 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-glzl6_5265ed5b-66c3-43a0-b4ff-f8b70892d133/istio-proxy/0.log" Oct 31 00:53:33.159967 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:53:33.159878 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:53:33.791744 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:33.791716 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/1.log" Oct 31 00:53:34.386983 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:34.386958 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:53:35.485886 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:35.485859 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/0.log" Oct 31 00:53:36.071451 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:36.071420 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/0.log" Oct 31 00:53:37.045486 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:37.045458 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7mjdd_76eccd75-30b9-4a48-88ec-d7aca906afa4/main/0.log" Oct 31 00:53:37.470322 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:37.470283 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-r99g8_959c5b03-4058-432e-8769-91e97e5deed9/main/0.log" Oct 31 00:53:37.864318 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:37.864234 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5bfgv_354612ef-bb88-4c36-b5fe-caf0ca868ce7/main/0.log" Oct 31 00:53:38.077186 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:38.077156 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp_8d2df4b9-ac3f-474f-a0f6-788c22573929/main/0.log" Oct 31 00:53:49.532509 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:49.532479 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_kserve-controller-manager-776cc66d47-jr6nm_130cd323-4efe-41e8-878a-d3b25ea31917/manager/0.log" Oct 31 00:53:50.227675 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:50.227644 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:53:50.473838 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:50.473807 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-75bd89fdfc-sxsw6_b9dad48a-5c54-4db3-a98e-581219b1046c/server/0.log" Oct 31 00:53:50.673942 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:50.673860 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_odh-model-controller-769875db56-mpwvn_6124af8d-fa59-479c-b6c0-561a759e7e71/manager/0.log" Oct 31 00:53:50.912719 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:50.912688 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_usage-logs-tenancy-proxy-9c5d49bcd-4w2nk_5b700d0c-b689-4767-9b25-aa14f4131036/proxy/0.log" Oct 31 00:53:51.497411 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:51.497383 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-8598cb9b7-rptl2_263c44c2-b7db-4633-b0ee-653c5a35d1af/maas-api/0.log" Oct 31 00:53:53.088716 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:53.088666 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_postgres-7cdbf6dc44-ngrsz_4a721706-8e9c-454a-81d0-19e4d564eebf/postgres/0.log" Oct 31 00:53:56.435601 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:56.435568 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qzxwc_de47760d-2eff-44ce-8aa5-9d82b9842a65/discovery/0.log" Oct 31 00:53:56.650257 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:56.650232 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-m2xt8_6521afb3-77b6-4c63-a867-e15f132b1acf/kube-auth-proxy/0.log" Oct 31 00:53:57.834491 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:57.834458 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/1.log" Oct 31 00:53:58.026227 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:58.026195 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-glzl6_5265ed5b-66c3-43a0-b4ff-f8b70892d133/istio-proxy/0.log" Oct 31 00:53:58.826964 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:58.826924 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/1.log" Oct 31 00:53:59.405615 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:53:59.405581 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:54:00.371129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:00.371094 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-7b998b79d9-7mjdd_76eccd75-30b9-4a48-88ec-d7aca906afa4/main/0.log" Oct 31 00:54:00.743107 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:00.743078 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-r99g8_959c5b03-4058-432e-8769-91e97e5deed9/main/0.log" Oct 31 00:54:01.132977 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:01.132893 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_echo-simulator-kserve-64b6f8f96c-5bfgv_354612ef-bb88-4c36-b5fe-caf0ca868ce7/main/0.log" Oct 31 00:54:01.324725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:01.324690 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-5dwkp_8d2df4b9-ac3f-474f-a0f6-788c22573929/main/0.log" Oct 31 00:54:03.161017 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:54:03.160980 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:54:33.159872 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:54:33.159830 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:54:42.938553 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:42.938516 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:54:43.216817 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:43.216726 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:54:43.966516 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:43.966483 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:54:44.154644 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:54:44.154614 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:55:03.161439 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:55:03.161404 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:55:33.159646 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:55:33.159604 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:55:35.295446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:35.295410 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:55:35.580488 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:35.580407 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-7b8bb6f8d-hb7vk_2cccc0c2-9727-4cd8-b458-87d18e8caf72/manager/0.log" Oct 31 00:55:36.462225 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.462198 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:55:36.651066 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.651035 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:55:36.984835 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.984797 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbcn2/must-gather-bpf7c"] Oct 31 00:55:36.985214 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985202 6234 cpu_manager.go:414] "containerMap: removing container" podUID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" containerName="payload-processing" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985215 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" containerName="payload-processing" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985234 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3c4caced-6f6c-44fb-8907-9c71d949240b" containerName="payload-pre-processing" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985239 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c4caced-6f6c-44fb-8907-9c71d949240b" containerName="payload-pre-processing" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985252 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985257 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985270 6234 cpu_manager.go:414] "containerMap: removing container" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.985279 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985274 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985289 6234 cpu_manager.go:414] "containerMap: removing container" podUID="3513c23e-0879-4525-91dd-62acc9c74136" containerName="maas-api" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985293 6234 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3513c23e-0879-4525-91dd-62acc9c74136" containerName="maas-api" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985349 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985360 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3513c23e-0879-4525-91dd-62acc9c74136" containerName="maas-api" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985370 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4cb5df6-9025-4b06-af6b-1c5fb50d6b8c" containerName="payload-processing" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985377 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c4caced-6f6c-44fb-8907-9c71d949240b" containerName="payload-pre-processing" Oct 31 00:55:36.985507 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.985497 6234 memory_manager.go:365] "RemoveStaleState removing state" podUID="8139042a-62c7-4354-8c2a-956c24fd5e01" containerName="istio-proxy" Oct 31 00:55:36.988446 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.988430 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:36.993757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.993733 6234 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-bbcn2\"/\"default-dockercfg-qbh96\"" Oct 31 00:55:36.993757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.993749 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"openshift-service-ca.crt\"" Oct 31 00:55:36.993757 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:36.993756 6234 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-bbcn2\"/\"kube-root-ca.crt\"" Oct 31 00:55:37.055851 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.055816 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-52rg8\" (UniqueName: \"kubernetes.io/projected/778fac92-3dcd-4592-9729-958183f5e2c1-kube-api-access-52rg8\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.056026 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.055927 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/778fac92-3dcd-4592-9729-958183f5e2c1-must-gather-output\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.064628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.064596 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/must-gather-bpf7c"] Oct 31 00:55:37.157274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.157240 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-52rg8\" (UniqueName: \"kubernetes.io/projected/778fac92-3dcd-4592-9729-958183f5e2c1-kube-api-access-52rg8\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.157444 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.157304 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/778fac92-3dcd-4592-9729-958183f5e2c1-must-gather-output\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.157622 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.157605 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/778fac92-3dcd-4592-9729-958183f5e2c1-must-gather-output\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.176258 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.176226 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-52rg8\" (UniqueName: \"kubernetes.io/projected/778fac92-3dcd-4592-9729-958183f5e2c1-kube-api-access-52rg8\") pod \"must-gather-bpf7c\" (UID: \"778fac92-3dcd-4592-9729-958183f5e2c1\") " pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.303129 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.303045 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" Oct 31 00:55:37.586946 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.586922 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/must-gather-bpf7c"] Oct 31 00:55:37.589414 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:55:37.589384 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod778fac92_3dcd_4592_9729_958183f5e2c1.slice/crio-8ba477e95429bb23a58d606c69a508084b102dc2b4e0b48b9d1a3c3c4a8a1f24 WatchSource:0}: Error finding container 8ba477e95429bb23a58d606c69a508084b102dc2b4e0b48b9d1a3c3c4a8a1f24: Status 404 returned error can't find the container with id 8ba477e95429bb23a58d606c69a508084b102dc2b4e0b48b9d1a3c3c4a8a1f24 Oct 31 00:55:37.591263 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:37.591248 6234 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider Oct 31 00:55:38.588244 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:38.588210 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" event={"ID":"778fac92-3dcd-4592-9729-958183f5e2c1","Type":"ContainerStarted","Data":"516fcf5f931bd157d6e0a277e61e2996d0df8de0f4678671c4f27184119171c8"} Oct 31 00:55:38.588571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:38.588260 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" event={"ID":"778fac92-3dcd-4592-9729-958183f5e2c1","Type":"ContainerStarted","Data":"15744ed2ea06b33ac73304567c889b20dd6789634cb410614dd13f2601870e74"} Oct 31 00:55:38.588571 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:38.588273 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" event={"ID":"778fac92-3dcd-4592-9729-958183f5e2c1","Type":"ContainerStarted","Data":"8ba477e95429bb23a58d606c69a508084b102dc2b4e0b48b9d1a3c3c4a8a1f24"} Oct 31 00:55:38.611247 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:38.611139 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bbcn2/must-gather-bpf7c" podStartSLOduration=1.8545956160000001 podStartE2EDuration="2.611120237s" totalImagesPullingTime="756.524621ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-10-31 00:55:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:38.611119502 +0000 UTC m=+3660.644215573" watchObservedRunningTime="2026-10-31 00:55:38.611120237 +0000 UTC m=+3660.644216308" Oct 31 00:55:40.243640 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:40.243606 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-45ssd_4285655f-2bda-4d2b-87f5-b9506c3fdb51/global-pull-secret-syncer/0.log" Oct 31 00:55:40.666796 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:40.666695 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-dbh42_5bdcd7f1-7d32-40eb-8daf-8c83e77e8296/konnectivity-agent/0.log" Oct 31 00:55:40.746436 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:40.746408 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-130-152.us-east-2.compute.internal_4e44186c11950043d5117909b34eae0d/haproxy/0.log" Oct 31 00:55:48.763805 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.763636 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/alertmanager/0.log" Oct 31 00:55:48.812320 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.812289 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/config-reloader/0.log" Oct 31 00:55:48.844221 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.844190 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/kube-rbac-proxy-web/0.log" Oct 31 00:55:48.881807 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.881740 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/kube-rbac-proxy/0.log" Oct 31 00:55:48.930976 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.930940 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/kube-rbac-proxy-metric/0.log" Oct 31 00:55:48.961162 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:48.961135 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/prom-label-proxy/0.log" Oct 31 00:55:49.006195 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.006142 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_18dfcc6b-111a-4d7b-ad7e-8bd92264af58/init-config-reloader/0.log" Oct 31 00:55:49.103871 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.099429 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bnpd7_46826886-bbcb-4f1e-ae2a-94f7feb26d76/kube-state-metrics/0.log" Oct 31 00:55:49.124454 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.124388 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bnpd7_46826886-bbcb-4f1e-ae2a-94f7feb26d76/kube-rbac-proxy-main/0.log" Oct 31 00:55:49.158459 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.158373 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_kube-state-metrics-5dc8d7d89b-bnpd7_46826886-bbcb-4f1e-ae2a-94f7feb26d76/kube-rbac-proxy-self/0.log" Oct 31 00:55:49.382818 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.382715 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2kgm_d775e8d4-9d4a-4779-a5c5-4442e8d3a306/node-exporter/0.log" Oct 31 00:55:49.408605 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.408575 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2kgm_d775e8d4-9d4a-4779-a5c5-4442e8d3a306/kube-rbac-proxy/0.log" Oct 31 00:55:49.438262 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.438214 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-q2kgm_d775e8d4-9d4a-4779-a5c5-4442e8d3a306/init-textfile/0.log" Oct 31 00:55:49.600339 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.600310 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfc9n_b9215d31-9f16-4011-a873-2995f3e65cf8/kube-rbac-proxy-main/0.log" Oct 31 00:55:49.636375 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.636300 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfc9n_b9215d31-9f16-4011-a873-2995f3e65cf8/kube-rbac-proxy-self/0.log" Oct 31 00:55:49.669045 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:49.669013 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-jfc9n_b9215d31-9f16-4011-a873-2995f3e65cf8/openshift-state-metrics/0.log" Oct 31 00:55:50.040292 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.040253 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4d947c48-bmwpb_9c881400-6424-435d-9c12-2fc6ba5d25f4/telemeter-client/0.log" Oct 31 00:55:50.073377 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.073325 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4d947c48-bmwpb_9c881400-6424-435d-9c12-2fc6ba5d25f4/reload/0.log" Oct 31 00:55:50.104090 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.104059 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-6c4d947c48-bmwpb_9c881400-6424-435d-9c12-2fc6ba5d25f4/kube-rbac-proxy/0.log" Oct 31 00:55:50.360072 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.359997 6234 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn"] Oct 31 00:55:50.367808 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.367762 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.376836 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.376027 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn"] Oct 31 00:55:50.450600 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.450561 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xj4mn\" (UniqueName: \"kubernetes.io/projected/e61a160a-b5c3-4559-8e57-548ca19d862d-kube-api-access-xj4mn\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.450840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.450622 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-sys\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.450840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.450683 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-lib-modules\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.450840 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.450800 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-podres\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.451049 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.450841 6234 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-proc\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552246 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552199 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-podres\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552280 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-proc\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552310 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xj4mn\" (UniqueName: \"kubernetes.io/projected/e61a160a-b5c3-4559-8e57-548ca19d862d-kube-api-access-xj4mn\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552321 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-podres\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552349 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-sys\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552413 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-proc\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552423 6234 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-lib-modules\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552482 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-sys\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.552710 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.552501 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/e61a160a-b5c3-4559-8e57-548ca19d862d-lib-modules\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.568705 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.568669 6234 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xj4mn\" (UniqueName: \"kubernetes.io/projected/e61a160a-b5c3-4559-8e57-548ca19d862d-kube-api-access-xj4mn\") pod \"perf-node-gather-daemonset-xg6rn\" (UID: \"e61a160a-b5c3-4559-8e57-548ca19d862d\") " pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.692670 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.692625 6234 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:50.991950 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:50.991852 6234 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn"] Oct 31 00:55:50.994665 ip-10-0-130-152 kubenswrapper[6234]: W1031 00:55:50.994630 6234 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pode61a160a_b5c3_4559_8e57_548ca19d862d.slice/crio-446fb7de51d40f677060dff4b5665599d0e21d5f7c01087406e0d442f5f09a30 WatchSource:0}: Error finding container 446fb7de51d40f677060dff4b5665599d0e21d5f7c01087406e0d442f5f09a30: Status 404 returned error can't find the container with id 446fb7de51d40f677060dff4b5665599d0e21d5f7c01087406e0d442f5f09a30 Oct 31 00:55:51.725568 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:51.725527 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" event={"ID":"e61a160a-b5c3-4559-8e57-548ca19d862d","Type":"ContainerStarted","Data":"3a1b28f99c330e5e06381926ca7fcb559aea810b4cb7eb4a4142000f95e4f0f3"} Oct 31 00:55:51.726089 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:51.725586 6234 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" event={"ID":"e61a160a-b5c3-4559-8e57-548ca19d862d","Type":"ContainerStarted","Data":"446fb7de51d40f677060dff4b5665599d0e21d5f7c01087406e0d442f5f09a30"} Oct 31 00:55:51.726089 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:51.725710 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:51.746667 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:51.745211 6234 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" podStartSLOduration=1.745191822 podStartE2EDuration="1.745191822s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-10-31 00:55:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-10-31 00:55:51.744894322 +0000 UTC m=+3673.777990393" watchObservedRunningTime="2026-10-31 00:55:51.745191822 +0000 UTC m=+3673.778287895" Oct 31 00:55:52.960577 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:52.960544 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_console-75bdffcc5c-glhxn_3a197b72-71ae-4bc5-9736-12979d594ab6/console/0.log" Oct 31 00:55:54.860363 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:54.860337 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dw7gl_14e77ba7-07ab-4933-9268-760aa9712cb8/dns/0.log" Oct 31 00:55:54.888207 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:54.888178 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-dw7gl_14e77ba7-07ab-4933-9268-760aa9712cb8/kube-rbac-proxy/0.log" Oct 31 00:55:55.128940 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:55.128859 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-q7jkq_78da32d6-6ce8-4fcc-93bb-11d27d3dbb6d/dns-node-resolver/0.log" Oct 31 00:55:55.843274 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:55.843247 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-4qqrz_827a07c8-e408-4526-a58b-4f7982a7dcaa/node-ca/0.log" Oct 31 00:55:57.314657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:57.314627 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qzxwc_de47760d-2eff-44ce-8aa5-9d82b9842a65/discovery/0.log" Oct 31 00:55:57.745068 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:57.745041 6234 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-bbcn2/perf-node-gather-daemonset-xg6rn" Oct 31 00:55:57.887037 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:57.887010 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-m2xt8_6521afb3-77b6-4c63-a867-e15f132b1acf/kube-auth-proxy/0.log" Oct 31 00:55:58.096657 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.096572 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/0.log" Oct 31 00:55:58.097114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.097095 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/1.log" Oct 31 00:55:58.126114 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.126084 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-glzl6_5265ed5b-66c3-43a0-b4ff-f8b70892d133/istio-proxy/0.log" Oct 31 00:55:58.291738 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.291703 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/0.log" Oct 31 00:55:58.291920 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.291753 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/1.log" Oct 31 00:55:58.412323 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:55:58.412235 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:56:00.925442 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:00.925408 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-5sqx2_32b07947-679b-40bb-849d-10815894262d/serve-healthcheck-canary/0.log" Oct 31 00:56:01.095629 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.095595 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qzxwc_de47760d-2eff-44ce-8aa5-9d82b9842a65/discovery/0.log" Oct 31 00:56:01.127248 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.127213 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-m2xt8_6521afb3-77b6-4c63-a867-e15f132b1acf/kube-auth-proxy/0.log" Oct 31 00:56:01.251973 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.251938 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/0.log" Oct 31 00:56:01.252185 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.252166 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/1.log" Oct 31 00:56:01.272080 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.272050 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-glzl6_5265ed5b-66c3-43a0-b4ff-f8b70892d133/istio-proxy/0.log" Oct 31 00:56:01.357529 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.357493 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/1.log" Oct 31 00:56:01.357730 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.357683 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/0.log" Oct 31 00:56:01.422704 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.422676 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log" Oct 31 00:56:01.935725 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.935699 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vscwx_58a99b61-5c4d-41c8-9b7a-172aa3edc72b/kube-rbac-proxy/0.log" Oct 31 00:56:01.980597 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:01.980567 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vscwx_58a99b61-5c4d-41c8-9b7a-172aa3edc72b/exporter/0.log" Oct 31 00:56:02.013628 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:02.013587 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-vscwx_58a99b61-5c4d-41c8-9b7a-172aa3edc72b/extractor/0.log" Oct 31 00:56:03.164624 ip-10-0-130-152 kubenswrapper[6234]: E1031 00:56:03.164568 6234 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-vscwx" containerName="extractor" Oct 31 00:56:04.651508 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.651467 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-fc89p_ea19c701-3688-49e2-8623-01fd1391aa28/manager/0.log" Oct 31 00:56:04.682694 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.682665 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-58f644b8bd-fc89p_ea19c701-3688-49e2-8623-01fd1391aa28/copy-manifests/0.log" Oct 31 00:56:04.715012 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.714975 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-qzxwc_de47760d-2eff-44ce-8aa5-9d82b9842a65/discovery/0.log" Oct 31 00:56:04.728058 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.728029 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-b8699df9-m2xt8_6521afb3-77b6-4c63-a867-e15f132b1acf/kube-auth-proxy/0.log" Oct 31 00:56:04.843168 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.843134 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/1.log" Oct 31 00:56:04.843331 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.843188 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-br6t8_46b6beec-0a69-4aa8-b4ab-1dec71762321/istio-proxy/0.log" Oct 31 00:56:04.858289 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.858260 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-glzl6_5265ed5b-66c3-43a0-b4ff-f8b70892d133/istio-proxy/0.log" Oct 31 00:56:04.936423 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.936383 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/1.log" Oct 31 00:56:04.936845 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.936821 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-8b64f9ff9-znrn9_758f2634-3775-48c4-a5c6-e1eab5e9e48e/istio-proxy/0.log" Oct 31 00:56:04.999938 ip-10-0-130-152 kubenswrapper[6234]: I1031 00:56:04.999912 6234 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-55fbbbf9dc-k94mw_8e683085-dae1-46d7-8e0f-b04bdd08a7dc/payload-processing/0.log"